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PREFACE 


This  work  was  performed  for  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness,  Readiness  and  Training  Directorate  under  a  task  entitled  “Development  and 
Assessment  of  ADL  Prototypes.” 

Advanced  Distributed  Learning  (ADL)  developments  require  collaboration  with 
individuals  working  on  the  many  different  facets  of  military  training.  In  addition  and 
despite  ADL’s  initial  emphasis  on  military  training,  these  ADL  developments  increas¬ 
ingly  require  intensive  collaboration  with  people  outside  the  military  in  other  govern¬ 
ment,  industry,  and  educational  organizations. 
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I.  INTRODUCTION 


Military  training  means  definite  but  very  different  things  to  different  people.  To 
the  commander  of  a  military  unit,  it  means  exercising  troops  in  the  field  or  sailors  at  sea 
so  that  they  operate  as  an  integrated,  coordinated  unit.  To  military  personnel  managers,  it 
means  preparing  and  certifying  individuals  across  a  full  spectrum  of  occupational  spe¬ 
cialties  that  include  cooks,  dog  handlers,  tank  turret  repairers,  radar  technicians,  and 
fighter  pilots.  To  developers  and  providers  of  major  military  systems,  it  means  exercises 
performed  in  simulators  or  on  the  systems  themselves.  To  all  concerned,  it  means  pre¬ 
paring  individuals  from  a  civilian  society  to  perform  as  professional  military  personnel. 

Military  training  is  distinguished  from  other  forms  of  training  by  its  emphases  on 
discipline,  just-in-case  preparation,  and  the  training  of  collectives.  This  document  is 
intended  to  provide  a  general  system-wide  overview  that  will  provide  a  useful  perspective 
on  military  training  for  all  those  involved  in  developing  and  assessing  prototypes  for  the 
Advanced  Distributed  Learning  (ADL)  initiative. 


1-1 


II.  WHAT  IS  UNIQUE  ABOUT  MIUITARY  TRAINING? 


A.  TRAINING  AS  DISCIPLINE 

Military  training  must  prepare  individuals  to  enter  into  harm’s  way  and  perform 
physically  and  mentally  demanding  tasks  at  the  highest  possible  levels  of  proficiency. 
This  requirement  may  be  the  defining  characteristic  of  military  training.  It  can  mean  the 
difference  between  life  and  death.  A  common  observation  among  tactical  analysts  and 
military  historians  is  that  the  greatest  harm  is  suffered  by  military  personnel  who  abandon 
their  tasks,  break,  and  run  under  the  pressures  of  combat  (e.g.,  du  Picq,  1880/1946; 
Keegan,  1993;  Gabriel  and  Metz,  1992).  For  these  reasons,  military  commanders  often 
view  training  as  discipline. 

Military  commanders  have  held  this  view  at  least  since  the  Persian  campaigns  of 
480-479  B.C.  in  Greece  (Dupuy  and  Dupuy,  1977).  In  those  campaigns,  the  Greeks 
relied  on  the  infantry  phalanx,  which  was  a  trained  body  of  soldiers  (hoplites)  arranged  in 
long  rows  that  varied  in  depth  from  about  8  to  16  men.  Training  emphasized  teamwork 
and  physical  conditioning.  Each  hoplite  had  to  learn  how  to  perform  precise  phalanx 
maneuvers.  This  was  not  simply  a  matter  of  executing  close-order  drill  on  parade  fields. 
Maintaining  a  controlled  front  while  advancing  over  mixed  terrain  was  critical  in  the  bro¬ 
ken  and  rocky  hills  of  Greece.  Each  hoplite  also  had  to  learn  to  use  his  weapons  profi¬ 
ciently,  to  stand  and  fight  without  exposing  his  comrades  to  flank  attack,  and  to  close 
ranks  quickly  when  others  fell. 

This  training  paid  off.  The  Persians  fought  with  greater  numbers  of  troops  and 
displayed  dash,  elan,  and  superior  generalship,  yet  they  were  continually  thwarted  by  the 
stubborn  discipline  and  training  of  the  hoplites.  In  the  battle  of  Platea,  the  Persians  shat¬ 
tered  their  land  forces  against  the  Greek  phalanxes,  losing  an  estimated  50,000  soldiers  to 
the  Greeks’  1,500.  Dupuy  and  Dupuy  (1977)  echo  points  made  earlier  by  Herodotus  and 
Thucydides,  noting  that  “the  battle  [of  Platea]  was  the  first  won  by  technical  superiority, 
in  the  first  clear-cut  example  of  the  value  of  superior  discipline  and  training”  (p.  28). 

Today’s  infusion  of  technology  into  nearly  every  aspect  of  military  operations  has 
significantly  altered  the  nature  of  military  engagements.  It  has  increased  the  complexity 


of  military  operations,  the  number  of  tasks  that  individuals  must  perform,  and  the  demand 
for  knowledge  and  skill  among  military  personnel  (Binkin,  1986;  Technology  for  the  U.S. 
Navy  and  Marine  Corps,  1997).  The  speed  and  mobility  of  modern  military  operations, 
the  lethality  and  long  reach  of  modern  weapons,  and  the  requirements  in  modern  doctrine 
for  dispersion  and  rapid  composition  of  forces  compound  this  complexity. 

To  some  degree,  technology  may  insulate  today’s  military  personnel  from  the 
exigencies  historically  faced  by  combatants.  Instead  of  charging  with  shield  and  sword, 
they  operate  increasingly  exotic  sensors  to  locate  the  enemy,  guide  robotic  vehicles 
toward  targets,  and  push  buttons  to  release  weapons.  They  contend  with  people  whom 
they  never  approach  in  person  and,  for  that  matter,  may  never  see.  Despite  their  technol¬ 
ogy,  however,  they  must  still  function  under  lethal  threat.  As  much  as  we  might  desire  the 
contrary,  modem  warfare  is  not  a  sanitized,  bloodless  affair.  Military  training  will  never 
escape  its  requirement  for  military  discipline. 

B.  JUST-IN-CASE  TRAINING 

Most  civilian  training  goes  to  considerable  lengths  to  avoid  just-in-case  training 
and  to  emphasize  just-enough  and  just-in-time  training.  Preparing  individuals  to  perform 
tasks  that  are  rarely— if  ever— required  on  the  job  is  viewed  as  unnecessary,  inefficient, 
wasteful,  and  generally  a  poor  idea.  Military  training  is  different.  It  must  prepare  indi¬ 
viduals  and  groups  of  individuals  for  armed  combat  engagements  that  we  hope  and  often 
expect  to  be  unnecessary.  In  this  sense,  military  training  is  fundamentally  just-in-case 
training,  treating  as  necessary  what  training  in  other  venues  may  view  as  extraneous. 

This  is  not  to  say  that  all  military  training  is  just-in-case  training.  Many  military 
tasks  are  not  performed  under  emergency  conditions,  allowing  time  for  just-in-time 
training  or  job-performance  aiding.  Still,  military  training  must  prepare  personnel  for 
armed  engagements— for  combat.  Succeeding  very  well  at  this  task  reduces  the  need  for 
it.  Knowing  that  they  face  a  well-trained,  well-prepared  enemy  has  caused  many  com¬ 
manders  to  withdraw  their  forces  from  the  field  before  combat  begins.  The  lethality  of 
modern  warfare  has  increased  the  deterrence  value  of  armed  forces  and,  thereby,  the 
value  of  training  for  combat,  but  it  may  also  reduce  the  likelihood  that  it  will  occur.  Most 
military  operations  in  recent  times  have  not  involved  combat.  Instead,  they  involve 
operations  such  as  peacekeeping,  peacemaking,  counterterrorism,  and  humanitarian 
relief.  In  recent  years,  only  about  10  percent  of  major  operations  undertaken  by  U.S. 
armed  forces  involved  armed  combat. 
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Some  just-in-case  training  is,  of  course,  properly  included  in  civilian  instruction. 
Police  departments,  fire  departments,  paramedics,  emergency  crews,  and  other  crews, 
groups,  and  teams  must  be  prepared  to  respond  appropriately  on  very  short  notice  to  rap¬ 
idly  developing  exigencies.  Combat  has  been  described  as  98  percent  boredom  and 
2  percent  sheer  terror.  Other  civilian  occupations  similarly  require  hours  of  watchful 
waiting  followed  by  rapid  transition  to  the  intense  concentrated  activity  and  teamwork 
needed  to  meet  emergencies.  These  transitions  and  the  just-in-case  training  needed  to 
prepare  for  them  are  of  sufficient  general  concern  that  they  have  been  studied  by  the 
National  Research  Council  (NRC)  (Huey  and  Wickens,  1993). 

A  dilemma  here  seems  peculiar  to  military  training.  Other  training,  including 
civilian  training  of  emergency  personnel  and  crews,  is  likely  to  be  assessed,  sooner  or 
later,  by  performance  in  “real-life”  situations.  Comprehensive  assessment  of  combat  skill 
is  both  vital  and  impossible.  Such  assessment  requires  the  exigent  pressures  of  wartime, 
which  exceed  anything  that  can  be  provided  in  peacetime.  Moreover,  a  single  combat 
engagement  undertaken  to  assess  the  capabilities  of  a  military  unit  would  be  insufficient 
because  of  the  decisive  and  unique  conditions  that  shape  military  operations.  We  would 
need  a  representative  sample  of  combat  engagements  to  complete  the  assessment.  The 
sample  would  have  to  be  stratified  to  account  for  various  combinations  of  air,  land,  and 
sea  assets;  different  conditions  of  weather,  terrain,  and  enemy  opposition;  and  different 
mission  intent.  Other  variables  could  be  added.  Staging  a  sample  of  wars  to  assess  com¬ 
bat  performance  and  effectiveness  rapidly  transcends  practicality,  to  say  nothing  of  ethi- 
cality. 

Hiller  (1987)  describes  this  issue  as  the  “criterion  problem”  of  combat  training. 
We  must  train  individuals  and  groups  of  individuals  without  the  benefit  of  true  criterion 
testing.  Given  the  destructive  and  horrific  nature  of  modem  combat,  we  can  only  be  glad 
the  problem  exists.  Still,  the  issue  remains  vital.  Unless  or  until  deep-seated  propensities 
in  human  nature  can  be  reversed,  provisions  for  the  defense  of  human  societies  will  con¬ 
tinue  to  be  essential  for  their  survival.  Societies  must  provide  just-in-case  combat  training 
for  their  armed  forces,  with  some  assurance  that  their  forces  — and  not  others  — will  pre¬ 
vail  should  such  operations  become  necessary.  How  to  assess  the  preparedness  of  mili¬ 
tary  forces  for  combat  engagements  presents  a  compelling  and  nationally  significant 
challenge  for  students  of  human  behavior. 
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C.  TRAINING  OF  COLLECTIVES 


Much  military  training  emphasizes  collectives:  crews,  groups,  teams,  and  units. 
Such  an  emphasis  may  be  reasonable  for  military  and  nonmilitary  training.  Cannon- 
Bowers,  Oser,  and  Flanagan  (1992),  in  their  review  of  collectives  in  industry  and  busi¬ 
ness,  report  a  clear  “consensus  among  those  who  study  industrial  and  organizational 
behavior  that  work  groups  are  the  cornerstone  of  modern  American  industry”  (p.  355). 
However,  collective  training  is  uniquely  emphasized  by  military  organizations.  An 
increased  concern  with  collective  training  has  been  growing  in  civilian  circles  (e.g., 
Swezey  and  Salas,  1992),  but  it  has  yet  to  match  the  concentrated  intensity  it  receives  in 
military  training. 

Collectives  are  certainly  characteristic  of  military  operations.  It  is  difficult  to 
think  of  any  military  operation  that  does  not  involve  the  performance  of  a  crew,  group, 
team,  or  unit.  For  that  matter,  many  military  organizations  may  not  consider  the  training 
of  individuals  as  a  training  issue  at  all— but  as  a  personnel  issue.  In  this  perspective, 
“training”  is  what  military  commanders  do  to  prepare  the  units  they  command  (the  units 
whose  performance  is  their  primary  responsibility)  to  perform  the  missions  to  which  they 
may  be  assigned.  Individual  “school-house”  training  to  certify  members  of  their  units  as 
tank  turret  repairers,  clerks,  machine  gunners,  torpedo  technicians,  and  so  forth  is  a  sepa¬ 
rate,  personnel  supply  activity.  It  is  to  be  performed  by  those  whose  job  it  is  to  provide 
units  with  qualified  individuals  to  fill  personnel  “slots.”  In  this  sense,  individual  training 
is  generally  viewed  not  as  an  investment;  rather,  it  is  viewed  as  an  infrastructure  cost, 
lumped  together  with  the  costs  of  transporting  personnel  from  one  unit  to  another, 
providing  for  their  hospitalization,  or  even  incarcerating  them  in  stockades  — a  cost  that  is 
to  be  avoided  or,  at  best,  minimized. 

Many  commentators  hold  this  view  of  training  as  primarily  and  perhaps  solely  as 
collective  preparation  on  military  operations.  It  is  reflected  by  much  that  is  written  about 
military  training  (e.g.,  Collins,  1978).  Overall,  the  concern  with  collective  training  and 
performance  is  not  exclusive— in  calm  moments,  most  military  commanders  recognize 
the  necessity  and  value  of  individual  training— but,  in  the  military,  collective  training  is 
emphasized  to  an  extent  rarely  found  in  nonmilitary  venues. 
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III.  RELEVANCE  OF  MILITARY  TRAINING 


Why  should  those  who  are  interested  in  civilian  training  attend  to  military 
training?  It  does  after  all  have  the  unique  emphases  discussed  previously,  but  these  are 
differences  of  emphasis.  Issues  of  discipline  to  do  the  work  needed,  just-in-case  prepara¬ 
tion  for  emergencies,  and  crew,  group,  team,  and  unit  training  are  also  found  in  civilian 
training  circles.  Aside  from  these  issues,  other  reasons  exist  for  those  responsible  for  the 
design,  development,  and  delivery  of  training  to  become  familiar  with  military  efforts  to 
do  the  same.  Among  these  reasons  are  those  that  stem  from  the  magnitude  of  military 
research  and  development  (R&D)  in  training  (particularly  R&D  on  technology  applied  to 
training),  specific  products  of  military  training,  practices  for  designing  and  developing 
military  training,  and  procedures  for  assessing  military  training  programs,  especially 
those  concerned  with  cost  and  effectiveness. 

A.  RESEARCH  AND  DEVELOPMENT  (R&D) 

Many  techniques  and  technologies  developed  for  military  training  are  also  appli¬ 
cable  in  civilian  sectors.  The  U.S.  Department  of  Defense  (DoD)  spends  hundreds  of 
millions  of  dollars  each  year  to  train  the  2.3  million  members  of  its  active  and  reserve 
armed  forces  and  its  800,000  civilian  employees.  Since  the  1950s,  it  has  spent 
$150  million  to  $250  million  each  year  on  R&D  in  education,  training,  training  devices, 
and  simulators.  DoD  maintains  what  may  be  the  largest,  most  extensive  training  opera¬ 
tion  in  history,  accompanied  by  what  may  be  the  largest,  most  extensive  training  R&D 
effort  ever  undertaken.  It  would  be  remarkable  if  all  this  activity  did  not  yield  some  tech¬ 
niques  and  technologies  of  general  interest  and  applicability  beyond  the  military. 

Defense-sponsored  R&D  is  notable  for  other  reasons.  This  activity  is  not  intended 
to  achieve  proprietary  advantage,  as  it  is  frequently  expected  to  do  in  the  private  sector. 
For  this  reason,  DoD  R&D  findings  are  as  freely  available  for  advancing  the  state  of  art 
and  practice  in  civilian  training  as  they  are  for  military  education— aside  from  issues  of 
classification  and  security.  The  impact  of  these  findings  in  the  United  States  has  been 
substantial,  as  Ellis  (1986)  documented,  but  it  is  often  indirect.  The  findings  are  likely  to 
be  disseminated  in  technical  reports  and  made  available  only  through  the  Defense 
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Technical  Information  Center  (DTIC)  or  the  National  Technical  Information  System 
(NTIS).  Defense  researchers  may  publish  their  findings  in  refereed  journals  but  only  later 
and  usually  with  few  incentives  other  than  personal  satisfaction  for  doing  so. 

Still,  the  transfer  of  DoD  R&D  products  to  civilian  practice  occurs  frequently, 
thereby  indicating  their  relevance  to  nonmilitary  instruction.  Such  “technology  transfer” 
can  include  different  devices  [e.g.,  the  aircraft  simulators  used  to  train  military  pilots, 
training  packages  that  constitute  programs  of  instruction  in  various  topics  of  interest,  and 
techniques  such  as  “intelligent”  computer-based  instruction  (CBI)  or  computer-based 
adaptive  testing]. 

Some  training  devices  have  been  transferred,  usually  as  surplus,  to  nonmilitary 
use,  but  this  form  of  transfer  is  infrequent.  There  has  been  considerable  interest  in 
adapting  military  training  packages  for  application  in  civilian  education  and  training 
institutions.  This  interest  does  not  seem  unreasonable.  As  the  next  section  suggests,  many 
military  training  programs,  packages,  simulations,  and  other  materials  prepared  for  mili¬ 
tary  training  could  find  direct  application  in  the  civilian  sector.  However,  it  is  the  transfer 
of  techniques  developed  by  DoD  R&D  funding— rather  than  devices  or  training  pack¬ 
ages— that  appears  to  be  most  common. 

This  sort  of  transfer  has  occurred  to  an  extent  rarely  noted  or  appreciated.  Fletcher 
and  Rockway  (1986)  reviewed  the  history  of  military  contributions  to  technology-based 
instruction.  They  suggested  that  DoD  funded  much,  and  perhaps  most,  of  the  early  work 
on  programmed  instruction,  computer-assisted  instruction  (CAI),  flight  simulation,  com¬ 
puter-managed  instruction  (CMI),  computer-based/adaptive  testing,  instructional  systems 
design,  interactive  videodisc  instruction,  and  intelligent  tutoring  systems.  The  U.S.  Con¬ 
gressional  Office  of  Technology  Assessment  (OTA)  reviewed  these  contributions  in  1988 
and  reported  that: 

The  military  has  been  a  major,  and  occasionally  the  major  player  in 
advancing  the  [educational  technology]  state  of  the  art.  Implications  for 
education  in  the  civilian  sector  are  clear.  Computers  would  probably  have 
found  their  way  into  classrooms  sooner  or  later.  But  without  [military 
support  of]  work  on  PLATO,  the  IBM  1500  system,  computer-based 
equipment  simulation,  intelligent  instructional  systems,  videodisc 
applications,  and  research  on  cognition,  it  is  unlikely  that  the  electronic 
revolution  in  education  would  have  progressed  as  far  and  as  fast  as  it  has 
( Power  On!,  1988,  p.  158). 
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B.  PRODUCTS  AND  PRACTICES  IN  MILITARY  TRAINING 
AND  EDUCATION 

A  considerable  overlap  exists  between  the  content  of  military  training  and  the 
content  of  training  found  elsewhere.  Table  III- 1  shows  the  diversity  of  enlisted  military 
careers.  The  table  lists  the  12  Standard  Occupational  Classifications  (SOCs)  for  enlisted 
military  careers  and  the  more  descriptive  occupational  categories  associated  with  each  of 
them.  On  any  day,  about  85,000  enlisted  personnel  will  be  taking  residential  training 
intended  to  certify  them  for  a  military  occupational  classification  or  subclassification 
(. Military  Manpower  Training  Report,  1999).  Each  year  there  are  about  900,000  enlisted 
personnel  enrollments  in  training  courses  that  last  from  4  to  72  weeks  in  length. 


Table  MI-1.  Training  Required  for  Military  Occupational  Specialties 


Standard 

Occupational  Code 

Occupational  Specialty 

Weeks  of 
Formal 
Instruction 

Administrative 

Administrative  Support  Specialists 

6-10 

Computer  Systems  Specialists 

7-13 

Finance  and  Accounting  Specialists 

6-12 

Flight  Operations  Specialists 

7-14 

Legal  Specialists  and  Court  Reporters 

6-10 

Personnel  Specialists 

7-9 

Postal  Specialists 

3-4 

Preventive  Maintenance  Analysts 

4-15 

Recruiting  Specialists 

4-6 

Sales  and  Stock  Specialists 

6-7 

Supply  and  Warehousing  Specialists 

4-6 

Training  Specialists  and  Instructors 

2-14 

Transportation  Specialists 

6-9 

Combat  Specialty 

Artillery  Crew  Members 

10-14 

Combat  Engineers 

None 

Infantrymen 

7-8 

Special  Operations  Forces 

Up  to  72 

Tank  Crew  Members 

6-9 

Construction 

Building  Electricians 

8-12 

Construction  Equipment  Operators 

4-12 

Construction  Specialists 

5-8 

Plumbers  and  Pipe  Fitters 

8-12 
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Table  111-1.  Training  Required  for  Military  Occupational  Specialties  (Continued) 


Standard 

Occupational  Code 

Occupational  Specialty 

Weeks  of 
Formal 
Instruction 

Electronic  and 

Aircraft  Electricians 

18-25 

Electrical  Equipment 

Communication  Equipment  Repairers 

8-40 

Repair 

Computer  Equipment  Repairers 

25-35 

Electrical  Products  Repairers 

4-22 

Electronic  Instrument  Repairers 

15-30 

Photographic  Equipment  Repairers 

9-32 

Power  Plant  Electricians 

4-17 

Precision  Instrument  Repairers 

12-34 

Radar  and  Sonar  Equipment  Repairers 

20-30 

Ship  Electricians 

18-25 

Weapons  Maintenance  Technicians 

15-30 

Engineering,  Science, 

Air  Traffic  Controllers 

7-13 

and  Technical 

Chemical  Laboratory  Technicians 

2-13 

Communications  Equipment  Operators 

9-22 

Computer  Programmers 

10-13 

Emergency  Management  Specialists 

8-10 

Environmental  Health  and  Safety  Specialists 

11-19 

Intelligence  Specialists 

9-24 

Meteorological  Specialists 

7-18 

Non-Destructive  Testers 

9-13 

Ordnance  Specialists 

15-25 

Radar  and  Sonar  Operators 

7-12 

Radio  Intelligence  Operators 

17-24 

Space  Operations  Specialists 

17-30 

Surveying,  Mapping,  and  Drafting  Technicians 

9-31 

Health  Care 

Cardiopulmonary  and  Electroencephalograph 
(EEG)  Technicians 

26-30 

Dental  Specialists 

9-14 

Medical  Care  Technicians 

7-52 

Medical  Laboratory  Technicians 

12-36 

Medical  Record  Technicians 

6-18 

Medical  Service  Technicians  (“Medics”) 

16-54 

Optometric  Technicians 

9-13 

Pharmacy  Technicians 

12-17 

Physical  and  Occupational  Therapy  Specialists 

11-31 

Radiological  (X-Ray)  Technicians 

12-19 
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Table  111-1.  Training  Required  for  Military  Occupational  Specialties  (Continued) 


Standard 

Occupational  Code 

Occupational  Specialty 

Weeks  of 
Formal 
Instruction 

Human  Services 

Caseworkers  and  Counselors 

8-10 

Religious  Program  Specialists 

7-8 

Machine  Operation  and 

Compressed  Gas  Technicians 

14-19 

Precision  Work 

Dental  and  Optical  Laboratory  Technicians 

21-26 

Machinists 

10-12 

Power  Plant  Operators 

12-25 

Printing  Specialists 

8-20 

Survival  Equipment  Specialists 

6-12 

Water  and  Sewage  Treatment  Plant  Operators 

8-10 

Welders  and  Metal  Workers 

4-15 

Media  and  Public 

Audiovisual  and  Broadcast  Technicians 

7-52 

Affairs 

Broadcast  Journalists  and  Newswriters 

9-12 

Graphics  Designers  and  Illustrators 

12-14 

Interpreters  and  Translators 

7-52 

Musicians 

11-24 

Photographic  Specialists 

7-24 

Service 

Firefighters 

7-11 

Food  Service  Specialists 

9-14 

Law  Enforcement  and  Security  Specialists 

5-12 

Military  Police 

8-12 

Transportation  and 

Air  Crew  Members 

7-9 

Material  Handling 

Aircraft  Launch  and  Recovery  Specialist 

9-13 

Cargo  Specialists 

2-6 

Flight  Engineers 

17-24 

Petroleum  Supply  Specialists 

4-8 

Quartermasters  and  Boat  Operators 

6-22 

Seamen 

6-12 

Vehicle  Drivers 

7-8 

Vehicle  and  Machinery 

Aircraft  Mechanics 

3-17 

Mechanics 

Automotive  and  Heavy  Equipment  Mechanics 

8-29 

Divers 

5-13 

Heating  and  Cooling  Mechanics 

8-22 

Marine  Engine  Mechanics 

9-24 

Powerhouse  Mechanics 

12-24 

About  16  percent  of  the  total  enlisted  force  is  assigned  to  the  five  combat  spe¬ 
cialty  occupations  shown  in  Table  III- 1 ,  and  about  25  percent  of  the  people  who  receive 
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initial  skill  training  are  in  these  combat  occupations.  There  are,  of  course,  civilian  coun¬ 
terparts  or  closely  related  activities  for  many  of  the  jobs  performed  in  combat  specialty 
occupations,  but  about  84  percent  of  the  remaining  enlisted  force  receives  initial  skill 
training  and  later  experience,  including  on-the-job  and  advanced  skill  training,  in  occu¬ 
pations  with  fairly  direct  civilian  counterparts. 

Examination  of  occupational  preparation  for  officers  exhibits  similar  proportions. 
On  any  day,  about  9,000  officers  take  residential  training  intended  to  certify  them  for  a 
military  occupational  classification  or  subclassification  {Military  Manpower  Training 
Report,  1999).  Each  year,  about  56,000  officers  enroll  in  training  courses.  About 
28  percent  of  these  officers  prepare  for  combat  specialty  occupations,  leaving  72  percent 
who  prepare  for  occupations  with  fairly  direct  civilian  counterparts. 

This  is  not  to  suggest  that  individuals  in  these  other  combat  support  and  combat 
systems  support  occupations  are  released  from  the  demands  necessarily  imposed  by 
military  discipline.  Many  individuals  in  these  occupational  categories  have  been  and  will 
be  exposed  to  armed  combat.  For  instance,  medical  service  technicians— even  those  who 
declare  themselves  conscientious  objectors— have  been  recognized  and  highly  decorated 
for  their  distinguished,  courageous  service  in  combat.  However,  Table  III- 1  makes  the 
point  that  military  occupations  and  training  are  diverse  and  concern  much  that  is  of  gen¬ 
eral  interest  and  wide  applicability  elsewhere. 

In  addition  to  the  skill  training  requirements  suggested  by  Table  III- 1 ,  military 
organizations  throughout  history  have  established  and  maintained  activities  that  in  other 
venues  would  look  very  much  like  basic  education.  Besides  teaching  every  soldier  to 
swim,  the  Romans  made  sure  that  all  soldiers  could  read  and  write  in  Latin  (Gabriel  and 
Metz,  1992).  Feudal  lords  who  raised  regiments  in  the  16th  century  also  ensured  that  sol¬ 
diers  in  their  infantry  regiments  were  sufficiently  literate  to  read  written  orders  and 
administrative  directives  (Keegan,  1993).  As  recruiting  grew  difficult  during  World 
War  II  and  the  Korean  War,  the  U.S.  Army  and  Navy  enlisted  soldiers  and  sailors  who 
had  to  be— and  were— taught  to  read  and  write  (Fletcher,  Duffy,  and  Curran,  1977). 

Currently,  the  U.S.  armed  forces  do  not  recruit  individuals  who  are  illiterate,  but 
they  do  enlist  individuals  who  must  improve  their  verbal  and  quantitative  skills  to  work 
in  specific  occupational  categories.  The  Basic  Skills  Education  Program  (BSEP)  dis¬ 
cussed  by  Wilson,  Golas,  and  O’Neil  (2000)  provides  one  example  of  the  direct  transfer 
that  can  occur  between  the  military  and  civilian  sector.  The  Army  developed  BSEP  in  the 
1980s  to  improve  the  basic  skills  preparation  of  individuals  for  specific  Army 


III- 6 


occupational  categories.  These  occupational  categories  were  of  sufficient  general  interest 
that,  in  response  to  numerous  requests,  the  Army  arranged  for  the  preparation  and  distri¬ 
bution  of  BSEP  for  use  in  civilian  training. 

In  response  to  a  Congressional  inquiry,  Fletcher  et  al.  (1992)  reviewed 
4,644  technology-based  training  products  that  the  U.S.  armed  forces  produced  for  their 
own  use.  They  found  that  2,718  (about  58  percent)  of  these  products  were  candidates  for 
civilian  application.  The  “tail-to-tooth”  ratio  suggested  by  this  percentage  of  potentially 
applicable  course  material  is  not  particularly  different  from  similar  ratios  found  in  other 
countries.  This  high  proportion  of  potentially  applicable  training  packages  suggests  that 
military  and  civilian  training  share  many  objectives,  approaches,  and  strategies  in  com¬ 
mon  and  that  they  have  much  to  learn  from  each  other. 

To  retain  skilled  and  experienced  military  personnel,  military  training  organiza¬ 
tions  must  attend  to  the  care  of  their  families—  including  provisions  for  K-12  education. 
The  U.S.  Department  of  Defense  Education  Activity  (DoDEA)  provides  or  otherwise 
supports  schooling  for  military  dependents.  These  schools  have  operated  on  U.S.  military 
bases  and  in  about  14  countries  overseas  since  1946.  In  the  1997-1998  school  year,  the 
Department  of  Defense  Dependents  Schools  (DoDDS)  served  an  estimated  80,000  stu¬ 
dents  in  161  schools  and  one  community  college  overseas.  In  the  same  school  year,  the 
Department  of  Defense  Domestic  Dependent  Elementary  and  Secondary  Schools 
(DDESS)  served  an  estimated  33,000  students  in  70  schools  located  in  7  states  and  the 
Commonwealths  of  Puerto  Rico  and  Guam. 

Military  organizations  have  supported  higher  education  for  a  long  time.  In  addi¬ 
tion  to  off-duty  vocational-technical  courses,  the  U.S.  DoD  Voluntary  Education  program 
provides  undergraduate  and  graduate  educational  opportunities  to  U.S.  military  personnel 
worldwide.  Each  of  the  military  Services  provide  these  opportunities  through  their  local 
education  centers.  In  1998,  more  than  6,000  baccalaureate  degrees  and  4,000  graduate 
degrees  were  awarded  to  participants  in  the  DoD  Voluntary  Education  program.  In  addi¬ 
tion,  the  three  Service  Academies  provide  undergraduate  training  each  year  to  about 
12,000  students.  The  academies  provide  the  higher  education  needed  by  military  officers, 
prepare  individuals  for  specific  military  activities,  and  instill  commitment  to  duty,  honor, 
and  country— the  virtues  of  military  discipline  (Forsythe,  1992). 

Finally,  military  organizations  require  a  broad  range  of  knowledge  and  skills, 
many  of  which  are  available  from  graduate  study  at  civilian  institutions.  DoD  sponsors 
graduate  education  for  about  5,000  personnel  each  year  and  graduates  about 
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1,000  individuals  from  the  Uniformed  Services  University  of  the  Health  Sciences 
(USUHS).  DoD  also  provides  professional  military  education,  which  is  intended  to  pre¬ 
pare  officers  for  the  increasingly  complex  demands  of  the  military  profession  and  the 
responsibilities  they  must  assume  as  they  progress  in  their  military  careers.  Nearly  all 
military  organizations  maintain  command  and  staff  “colleges”  at  both  intermediate  and 
senior  levels.  Officers  specifically  selected  for  advancement  attend  these  colleges.  Post¬ 
graduate  education  of  this  sort  includes  specific  training  for  the  performance  of  large- 
scale  military  operations,  general  courses  in  the  history  and  technology  of  military 
operations,  and  wide-ranging  instruction  in  the  many  economic,  political,  and  social  fac¬ 
tors  involved  in  national  security. 

C.  DESIGN  AND  DEVELOPMENT  OF  MILITARY  TRAINING 

The  rapid  changes  in  technology,  tactics,  and  missions  that  are  characteristic  of 
today’s  military  operations  require  matching  agility  in  the  design  and  development  of 
training  and  education  programs.  These  changes  must  be  made  quickly  and  efficiently. 
The  U.S.  defense  community  recognized  this  need  early  and  began  to  apply  the  tech¬ 
niques  and  processes  of  systems  engineering  to  the  design  and  development  of  training. 
Systems  engineering  had  served  the  DoD  well  in  a  host  of  other  applications,  and  it  pro¬ 
vided  a  foundation  for  R&D  that  produced  Instructional  Systems  Development  (ISD) 
(Logan,  1979)  and,  more  recently,  Systems  Approach  to  Training  (SAT)  (Guptil,  Ross, 
and  Sorenson,  1995). 

ISD/SAT  approaches  apply  standard  systems  engineering  to  the  development  of 
instructional  programs.  They  extend  the  generic  systems  components  of  analysis,  design, 
production,  implementation,  and  evaluation  to  training  and  education.  ISD/SAT  com¬ 
bines  these  engineering  components  with  theories  of  learning  and  instruction  to  produce 
systematically  designed  and  effective  training  programs. 

•  Training  analysis  is  based  on  systematic  study  of  the  job  and  the  task(s)  to  be 
performed.  It  identifies  training  inputs  and  establishes  training  objectives  to 
be  accomplished  in  the  form  of  student  flow  and  in  the  knowledge,  skill,  and 
attitude  outcomes  to  be  produced  by  the  training. 

•  Training  design  devises  the  instructional  interactions  needed  to  accomplish 
the  training  objectives  identified  by  training  analysis.  It  also  selects  the 
instructional  approaches  and  media  used  to  present  these  interactions. 

•  Training  production  concerns  the  development  and  preparation  of 
instructional  materials.  These  materials  can  include  hardware  (e.g., 
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simulators),  software  (e.g.,  computer  programs  and  audiovisual  productions), 
and  databases  for  holding  information  (e.g.,  subject  matter  content  and  the 
performance  capabilities  of  military  systems  ranging  from  tanks  and 
airplanes  to  radios  and  parachutes). 

•  Training  implementation  concerns  the  appropriate  installation  of  training 
systems  and  materials  in  their  settings  and  attempts  to  ensure  that  they  will 
perform  as  designed. 

•  Training  evaluation  reflects  the  outcomes  of  the  training  against  its 
objectives.  It  determines  if  the  training  does  things  right  and  if  it  does  the 
right  things.  It  verifies  that  the  training  system  meets  its  objectives,  provides 
the  validation  that  meeting  these  objectives  prepares  individuals  to  perform 
targeted  tasks  or  jobs  better,  and  improves  the  operation  of  the  organization 
overall. 

Tennyson  and  Foshay  (2000)  discuss  these  processes  in  more  detail.  They  have 
become  as  widely  recommended  and  used  in  civilian  education  and  training  as  they  have 
in  the  military.  In  applying  them  to  the  preparation  for  military  operations,  training  man¬ 
agers  start  from  the  top,  with  the  missions  and  operations  that  military  organizations  at 
each  level  are  designed,  manned,  supplied,  and  expected  to  perform.  This  beginning  leads 
directly  to  the  identification  and  description  of  all  the  tasks  that  the  organization  (and  the 
men  and  women  in  it)  must  perform  to  accomplish  its  missions  and  operations  success¬ 
fully.  The  compilation  of  tasks  for  any  given  organization  can  be  described  as  its  mis¬ 
sion-essential  task  list  (METL).  A  military  organization’s  METL  forms  the  basis  for  its 
training  plans. 

Identification  of  the  METL  for  an  organization  is  only  part  of  the  story.  Trainers 
must  establish  baseline  conditions  under  which  each  mission-essential  task  must  be  per¬ 
formed  and  the  standard  of  performance  that  determines  successful  performance  of  the 
task.  This  combination  of  tasks,  conditions,  and  standards  provides  the  basis  for  much  of 
today’s  military  training.  The  training  readiness  of  individuals  and  their  organization  is 
determined  by  performance  on  mission-essential  tasks,  each  of  which  is  further  defined 
by  conditions  and  standards. 

Analogous  applications  involving  the  development  of  essential  tasks,  conditions, 
and  standards  and  their  use  in  assessing  the  organization’s  training  and  educational 
“readiness,”  or  likely  success  in  performing  its  functions,  are  easy  to  imagine.  However, 
this  extended  application  of  ISD/SAT  is  rarely  found  outside  the  military. 
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D.  ASSESSMENT  OF  MILITARY  TRAINING 


Many  individuals  in  the  military  and  civilian  training  communities  are  familiar 
with  Kirkpatrick’s  (1987)  steps  or  levels  of  evaluation.  These  levels  can  be  described 
roughly  (and  briefly)  as  shown  in  Table  III-2. 


Table  111-2.  Kirkpatrick’s  Four  Levels  of  Evaluation 


Level 

Description 

Evaluation  Issue 

1 

Surveys 

What  did  people  think  of  the  training? 

2 

Training  Outcome  Measures 

Did  the  training  achieve  its  objectives? 

3 

Transfer 

Did  job/work  performance  improve? 

4 

Benefits 

Did  organizational  performance  improve? 

These  four  steps  are  as  relevant  to  military  training  as  they  are  to  civilian  applica¬ 
tions,  and  they  are  applied  with  about  the  same  frequency  in  military  training  as  they  are 
in  civilian  training.  Many  assessments  of  military  training  stop  at  Level  1,  with  surveys  of 
the  trainees  and  instructors.  Some  proceed  to  Level  2,  with  end-of-course  measurement  of 
the  knowledge  and  skills  provided  by  the  training.  These  are  the  topics  of  reviews  such  as 
those  by  Orlanksy  and  String  (1977,  1979)  and  meta-analyses  such  as  those  by  Fletcher 
(1990).  A  few  assessments  of  military  training  extend  to  Level  3,  with  measurement  of 
on-the-job  performance  improvements;  however,  those  that  do  are  often  limited  to  opin¬ 
ion  surveys  of  peers  and  supervisors  and  rarely  involve  genuine  measurement  of  job 
performance. 

One  point  emphasized  by  Kirkpatrick’s  levels  is  that  training  — military  and  civil¬ 
ian— is  a  means  to  an  end.  Whatever  excellence  we  achieve  in  training  design,  develop¬ 
ment,  and  delivery  will  remain  irrelevant  if  the  end  it  serves  is  not  achieved.  Hence, 
Level  4. 

At  Level  4,  civilian  trainers  consider  issues  such  as  productivity  and  profitability. 
Military  trainers  must  consider  operational  effectiveness,  which  keys  on  combat  effec¬ 
tiveness.  Hiller’s  (1987)  criterion  problem  helps  explain  the  scarcity  of  Level  4  assess¬ 
ments  in  military  training  evaluations.  A  few  Level  4  evaluations  of  military  training 
have  been  undertaken  to  link  training  inputs  with  operational  effectiveness.  For  instance, 
Cavaluzzo  (1984)  found  that  a  1 -percent  increase  in  flying  hours  was  associated  with  a 
0.5-percent  decrease  in  average  bombing  miss  distance;  Hammond  and  Horowitz  (1990) 
found  that  a  decrease  of  10  percent  in  flying  hours  increased  the  probability  of  defeat  in 
air-to-air  combat  by  9.2  percent;  and  Holz,  O’Mara,  and  Keesling  (1994)  found  that  the 
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number  of  tank  miles  driven  was  correlated  with  offensive  mission  performance 
(r  =  0.68)  and  defensive  mission  performance  (r  =  0.80). 

With  today’s  mix  of  missions,  which  include  peacemaking,  peacekeeping,  coun¬ 
terterrorism,  and  humanitarian  relief,  the  inclusion  of  Level  4  assessments  for  military 
training  inputs  are  increasingly  feasible  as  these  missions  become  increasingly  common 
for  all  nations’  military  organizations.  They  should  be  pursued.  However,  the  heart  of 
military  capability  remains  effectiveness  in  armed  engagements.  Level-4  evaluation  of 
training  for  this  capability  will  remain  difficult  for  all  military  organizations. 

Kirkpatrick  and  others  who  are  concerned  with  the  evaluation  of  training  and  edu¬ 
cation  emphasize  instructional  effectiveness.  There  is  another  side  of  the  coin,  however. 
Assessments  are  performed  to  inform  decisions  — in  this  case,  decisions  about  training. 
The  hallmark  of  most,  perhaps  all,  decision  making  is  not  just  the  business  of  seeking 
improvement  but  also  determining  what  must  be  given  up  to  achieve  it.  Here,  we 
encounter  the  issues  not  only  of  effectiveness,  but  of  costs  and  cost  effectiveness. 

E.  COST,  EFFECTIVENESS,  AND  COST  EFFECTIVENESS 

Most  evaluations  of  new  instructional  approaches  are  performed  by  instructional 
researchers  and  innovators  who  simply  want  to  know  if  a  new  approach  works  and  if  it 
works  better  than  what  they  now  have.  For  managers  and  policymakers,  this  is  only  half 
the  story.  Increasing  effectiveness  is  important,  but  systems  are  closed  and  budgets  are 
limited.  Decisions  about  the  allocation  of  scarce  resources  among  competing  alternatives 
to  maximize  organizational  goals  and  objectives  are  a  perennial  necessity.  Such  decision 
making  is  as  common  in  military  training  as  it  is  elsewhere. 

The  cost  effectiveness  of  training  appears  to  be  more  frequently  and  openly  dis¬ 
cussed  in  the  military  than  in  the  civilian  area.  This  difference  may  be  associated  with  the 
nature  of  nonmilitary  training,  which  generally  occurs  in  the  profit-and-loss  world  of 
business  and  industry,  where  training  is  performed  to  achieve  proprietary  advantage. 
Military  training  is  supported  with  public  funding  and  is  intended  to  meet  national  objec¬ 
tives.  For  this  reason,  its  costs,  effectiveness,  and  findings  are  more  open  to  public  review 
and  discussion.  Certainly,  its  principles  and  lessons  learned  should  be  open  to  all.  This  is 
particularly  true  for  costs  and  effectiveness,  which  are  sensitive  topics  for  business  and 
industry.  Whatever  the  case,  most  freely  available  information  on  the  cost  effectiveness 
of  different  training  approaches  seems  to  emerge  from  military  applications. 
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Cost-effectiveness  considerations  may  highlight  one  dimension  of  difference 
between  education  and  training.  Education,  as  an  end  in  its  own  right— something  done 
for  its  own  sake— is  often  concerned  with  the  maximization  of  effectiveness  given  fixed 
costs.  Training,  as  a  means  to  an  end,  is  often  concerned  with  minimizing  costs  to  achieve 
a  given  level  of  effectiveness.  These  differing  approaches  to  questions  of  cost  effective¬ 
ness  appear  valid— but  only  up  to  a  point.  Educators  do  have  objectives,  and  educational 
decision  makers  do  have  an  interest  in  minimizing  costs  to  achieve  them.  Trainers  operate 
from  fixed  budgets.  They  have  an  interest  in  squeezing  the  most  instructional  effective¬ 
ness  they  can  from  these  budgets.  Military  trainers,  as  interested  as  they  must  be  in 
minimizing  costs,  increasingly  and  naturally  emphasize  effectiveness  rather  than  costs  as 
the  preparations  for  armed  combat  come  closer. 

Whatever  the  case,  those  whose  assessments  are  intended  to  inform  decisions 
concerning  the  adoption  of  training  programs  must  consider  costs  and  effectiveness.  Even 
researchers,  who  want  to  know  if  their  new  ideas  “work,”  must  consider  costs  if  they 
wish  to  make  a  difference  in  practice  and  influence  training  decision  makers.  Studies  that 
fail  to  consider  both  costs  and  effectiveness  are  insufficient  for  making  organizational 
decisions  and  setting  policies. 

Allocation  of  instructional  resources  or  expenditures  in  military  and  nonmilitary 
applications  is  based  on  economic  analysis,  a  technique  to  help  resource  allocation  deci¬ 
sions  attain  maximum  efficiency.  The  assumption  here  is  that  there  is  a  combination  of 
resources  that  maximizes  instructional  productivity  and  uses  minimum  resources.  Maxi¬ 
mum  instructional  efficiency  has  been  attained  when  the  combination  of  resources  that 
will  improve  instructional  productivity  without  increasing  instructional  costs  cannot  be 
found.  Economic  analysis  for  training  must  assess  the  repertoire  of  approaches  available 
to  achieve  maximum  instructional  efficiency. 

The  basic  idea  behind  economic  analysis  is  the  well-established  and  straight¬ 
forward  notion  that  there  is  no  free  lunch.  The  economic  analyst  looks  for  opportunities 
in  selecting  any  alternative.  This  suggests,  as  did  Okun  (1970)  when  he  articulated  the 
approach,  that  the  economic  analyst  pursues  a  marginalist  or  incrementalist  approach. 
The  economic  analyst  must  insist  on  knowing  how  much  is  gained  and  how  much  is 
given  up  by  doing  something  different. 

Economic  analysis  incorporates  the  concepts  of  cost-benefit  analysis  and  cost- 
effectiveness  analysis.  Cost-benefit  analysis  is  generally  used  to  determine  if  the  benefits 
of  projects  and  policies  outweigh  the  costs.  The  assumption  underlying  cost-benefit 


III- 12 


analysis  is  that  both  costs  and  benefits  are  measurable  in  the  same  units,  which  are  usu¬ 
ally  monetary  units  such  as  dollars.  This  commensurability  is  a  prerequisite  for  cost- 
benefit  analysis. 

The  full  benefits  of  instruction  may  be  impossible  to  assess  in  monetary  units, 
despite  heroic  attempts  by  economists  to  do  so.  Commensurability  that  adequately 
reflects  the  benefits  of  instruction  is  difficult  to  attain  in  comparisons  of  instructional 
alternatives  and  their  outcomes.  The  problem  is  particularly  difficult  in  light  of  Hiller’s 
criterion  problem  and  associated  inadequacies  in  our  measures  of  unit  readiness  to  per¬ 
form  military  operations. 

When  commensurability  is  lacking,  cost-effectiveness  analysis  is  used.  Again, 
costs  are  usually  expressed  in  monetary  units  such  as  dollars.  Benefits  (e.g.,  information 
retention;  the  productivity,  job  knowledge,  and  motivation  of  workers;  favorable  supervi¬ 
sor  ratings;  and  the  productivity  and  effectiveness  of  the  client  organization)  are  meas¬ 
ured  in  their  own  units,  usually  along  as  many  dimensions  as  possible  to  reflect  the  full 
spectrum  of  instructional  outcomes. 

In  performing  cost-effectiveness  analyses  for  instruction,  a  common  practice  is  to 
hold  either  costs  or  effectiveness  constant  and  observe  variations  in  the  other  variable 
across  the  alternatives  being  considered.  Often  the  variable  is  not  actually  held  constant 
but  is  simply  assumed  to  be  the  same  across  the  alternatives.  There  are  evaluations  in 
which  cost  of  competing  alternatives  is  implicitly  assumed  to  be  the  same,  and  there  are 
just  as  many  evaluations  in  which  the  effectiveness  of  competing  alternatives  is  implicitly 
assumed  to  be  the  same.  Frequently,  no  data  or  information  is  presented  to  support  the 
assumptions  of  equal  costs  or  equal  effectiveness,  and  decision  makers  must  take  it  on 
faith  that  the  assumptions  are  warranted. 

These  considerations  of  cost  and  effectiveness  lead  naturally  to  the  question  of 
what  can  be  done  in  real  practice.  One  necessity  in  measuring  the  costs  of  instruction  is  a 
practicable  list  or  “structure”  of  well-defined  cost  components.  These  have  been  called 
ingredients  by  Levin  (1983)  and  elements  by  Knapp  and  Orlansky  (1983).  The  list 
should,  for  any  instructional  alternative,  capture  all  the  components  needed  for  cost- 
effectiveness  comparisons  regardless  of  the  scope,  complexity,  or  technology  of  the 
alternative.  It  should  also  ensure  in  the  analysis  a  level  of  detail  that  clearly  identifies  the 
“cost  drivers”  — the  major  contributors  to  the  cost  of  the  alternative.  The  list  should  be 
usable  for  selecting,  planning,  assessing,  and  modifying  instructional  alternatives. 
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Four  general  categories  are  found  in  most  cost  models: 

•  R&D.  R&D  costs  consist  of  all  hardware,  software,  other  materials,  people,  and 
facilities  necessary  to  create,  test,  and  evaluate  an  instructional  approach. 

•  Initial  investment.  Initial  investment  costs  comprise  the  one-time  costs  of  procuring 
and  deploying  resources  in  the  quantities  needed  to  satisfy  anticipated  requirements 
for  an  instructional  approach. 

•  Operations  and  support  (O&S).  O&S  costs  include  those  needed  for  managing, 
operating,  and  maintaining  an  instructional  approach  after  it  has  been  implemented. 

•  Disposal  and  salvage.  Disposal  and  salvage  costs  comprise  the  one-time  costs  of 
removing  the  instructional  approach  from  operational  use. 

In  practice,  R&D  costs  are  set  aside  as  being  too  hard  to  determine,  too  difficult  to 
recover,  and  basically  “sunk.”  There  is  little  to  decide  about  the  resources  that  have  been 
expended.  Disposal  and  salvage  costs  are  relevant  in  several  areas,  but  these  costs  tend  to 
be  minor  in  training  programs.  They,  too,  are  usually  set  aside.  What  remain  are  initial 
investment  and  O&S  costs.  These  two  categories  of  costs  should  be  included  in  cost- 
effectiveness  analyses  intended  to  inform  about  choices  to  be  made  among  different 
instructional  approaches.  Knapp  and  Orlansky  (1983)  developed  a  comprehensive  cost 
element  model  for  evaluating  training  systems.  It  provides  a  common  basis  for  cost 
evaluations  and  cost  comparisons  of  training  programs  and  should  be  practicable  for 
training  applications  inside  and  outside  the  military. 

Basically,  three  questions  need  to  be  answered: 

1.  How  much  more  or  less  effectiveness  does  the  new  technology,  device,  or 
method  of  training  provide? 

2.  How  much  more  or  less  does  the  new  technology,  device,  or  method  of 
training  cost? 

3.  If  the  new  technology,  device,  or  method  improves  the  effectiveness  of 
training,  is  the  improvement  worth  its  cost? 

Kazanowski  (1968)  developed  a  standardized,  10-step  approach  to  cost- 
effectiveness  evaluations.  The  approach,  which  was  developed  for  selecting  weapons 
system  capabilities,  provides  a  foundation  for  evaluating  cost-effectiveness  in  instruction. 
Kazanowski’s  10  steps  applied  to  instruction  might  be  described  as  follows: 

•  Step  1:  Define  the  objectives.  Training  objectives  are  most  often  expressed 
in  terms  of  what  students  can  do  (skills),  what  they  must  know  (knowledge), 
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and/or  the  attitudes  they  must  possess  after  they  finish  the  instruction.  In 
training,  the  objectives  can  be  derived  directly  from  the  skills  and  knowledge 
required  to  perform  a  job.  In  the  absence  of  these  objectives,  relevant, 
systematic  design,  development,  implementation,  and  evaluation  of  the 
instruction  is  unlikely. 

•  Step  2:  Identify  the  mission  requirements.  The  instructional  system  must 
possess  productivity,  or  “pipeline,”  characteristics  defined  by  its  mission.  It 
must  be  able  to  churn  out  a  given  number  of  graduates  within  given  amount 
of  time.  The  instructional  mission  determines  these  pipeline  requirements. 

•  Step  3:  Develop  the  alternatives.  Once  the  instructional  objectives  for 
student  outcomes  have  been  defined  and  the  pipeline  requirements  of  the 
instructional  program  have  been  established,  locating  or  devising  alternative 
approaches  for  satisfying  these  objectives  and  requirements  is  appropriate. 
Generating  alternatives  is  a  critical  activity  and  requires  considerable 
imagination  and  creativity.  Tools  and  aids  that  help  are  available,  but  no  sure 
procedures  for  developing  comprehensive  sets  of  instructional  alternatives 
exist. 

•  Step  4:  Design  the  effectiveness  evaluation.  The  measures  to  be  examined 
by  an  effectiveness  evaluation  should  be  well  defined  and  evident.  These 
measures  should  follow  directly  from  the  instructional  objectives  and 
pipeline  requirements  established  for  the  program. 

•  Step  5:  Select  a  fixed  cost  or  fixed  effectiveness  approach.  As  discussed 
previously,  trainers  are  likely  to  hold  effectiveness  constant  and  try  to 
minimize  cost  to  achieve  that  level  of  effectiveness,  whereas  educators  are 
more  likely  to  hold  cost  constant  and  try  to  maximize  effectiveness  to  be 
obtained  for  that  cost.  It  is  possible  to  avoid  choosing  between  fixed  cost  and 
fixed  effectiveness  approaches  by  calculating  and  then  comparing  ratios  of 
cost  to  effectiveness  or  ratios  of  effectiveness  to  cost  for  the  various 
alternatives.  A  ratio  approach  of  this  sort  is  appropriate  only  when  cost  and 
effectiveness  that  are  increments  to  some  baseline  are  being  considered,  and, 
even  then,  significant  methodological  caution  is  advisable. 

•  Step  6:  Determine  the  capabilities  of  the  alternative  systems.  Once  the 
measures  and  evaluation  approach  has  been  settled,  it  is  time  to  gather  the 
data  and  proceed  with  the  analysis. 

•  Step  7:  Tabulate  the  alternatives  and  measures.  The  alternatives  are 
usually  tabulated  following  the  “northwest”  rule.  That  is  to  say,  the  measures 
are  listed  from  left  to  right  in  decreasing  priority,  and  the  alternatives  are 
listed  from  top  to  bottom  in  order  of  decreasing  apparent  value. 
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•  Step  8:  Analyze  the  merits  of  the  alternative  systems.  Decision  makers 
need  information.  Discussing  the  findings,  evaluating  the  strengths  and 
weaknesses  of  the  data  collection  procedures,  and  assessing  the  alternatives 
in  light  of  the  evaluation  should  be  pursued. 

•  Step  9:  Perform  a  sensitivity  analysis.  The  outcome  of  a  cost-effectiveness 
evaluation  may  or  may  not  be  sensitive  to  the  assumptions  on  which  it  is 
based.  Sensitivity  analyses  should  be  performed  to  assess  this  possibility. 

•  Step  10:  Document  the  bases  of  the  previous  nine  steps.  The  range  of 
issues  and  concerns  ensures  that  no  cost-effectiveness  evaluation  will  satisfy 
all  those  who  have  a  stake  in  its  outcomes,  but  it  can  and  should  be  explicit. 
Decision  makers  should  know  its  strengths  and  limitations.  Underlying 
models  of  cost  and  effectiveness  should  be  documented  so  that  decision 
makers  can  determine  how  relevant  any  analysis  is  to  their  concerns. 

F.  ENGINEERING  OF  INSTRUCTION 

Finally,  just  as  there  is  a  range  of  cost  measures  to  be  considered  in  instruction,  so 
is  there  a  range  of  effectiveness  measures.  The  end-of-instruction  test,  typically  used  to 
evaluate  the  effectiveness  of  instruction,  rarely  reflects  the  full  range  of  outcomes 
instructional  managers  and  other  decision  makers  expect  in  return  for  their  investment  in 
the  program.  These  outcomes  include  speed  of  response,  accuracy  of  response,  short-  and 
long-term  retention  of  performance  and  knowledge,  ability  to  transfer  performance  and 
knowledge  to  new  situations,  insight  and  the  ability  to  teach  others  what  was  learned, 
adherence  to  procedure,  and  motivation  to  improve  performance  and  knowledge  in  the 
subject  area.  Which  subset  of  these  objectives  ought  to  be  measured  depends  on  the 
intentions  of  training  decision  makers.  Different  measures  could  be  used  to  evaluate  the 
same  instructional  program  depending  on  these  intentions. 

Woolf  and  Regian  (2000)  further  discuss  the  engineering  of  instruction  and  its 
importance  in  designing  training  programs  that  are  aligned  with  the  needs  and  perceived 
utilities  of  any  organization  that  implements  them.  Rather  than  repeat  their  discussion 
here,  it  may  suffice  to  note  that  in  addition  to  defining  and  describing  an  adequate  model 
of  costs  in  the  assessment  of  training,  an  adequate  model  of  training  outcomes  should  be 
defined  and  described  in  assessing  the  contributions  of  any  training  program— military  or 
otherwise— to  its  sponsors  and  users. 
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IV.  VARIETIES  OF  MILITARY  TRAINING 


The  extent  and  diversity  of  military  training  make  it  difficult  to  discuss  coher¬ 
ently.  General  Paul  Gorman  suggested  a  robust  partitioning  of  military  training  that  con¬ 
tinues  to  find  wide  use.  As  shown  in  Table  IV-1,  Gorman’s  matrix  divides  military 
training  into  four  cells  keyed  to  whom  is  trained  (individuals  or  “collectives”)  and  where 
the  training  takes  place  (in  residence  or  in  operational  units).  In  DoD  organizations,  this 
structure  seems  to  evolve  almost  inevitably  as  roles  and  responsibilities  for  the  overall 
management  and  conduct  of  training  evolve. 


Table  IV-1.  Gorman’s  Matrix:  Components  of  Military  Training 


Where  Training  Takes  Place 

Who  is  Trained 

Residence 

Operational  Units 

Individuals 

Training  conducted  by  training  orga¬ 
nizations  to  develop  individual  skills 
and  knowledge  in  formally  convened, 
centralized  settings 

Training  conducted  by  operational 
units  to  develop  individual  skills  and 
knowledge  in  distributed  settings 

Collectives 

Training  to  achieve  crew,  team,  and 
unit  performance  standards  in  for¬ 
mally  convened,  centralized  settings 

Training  to  achieve  crew,  team,  and 
unit  performance  standards  in  opera¬ 
tional  units  and  other  distributed  set¬ 
tings 

A.  INDIVIDUAL  AND  COLLECTIVE  TRAINING 

Training  objectives  can  center  on  the  performance  of  either  individuals  or  teams. 
Most  research  and  studies  of  learning  and  teaching  center  on  individuals.  However,  as 
suggested  by  Cannon-Bowers,  Oser,  and  Flanagan  (1992),  among  others,  most  human 
activity  is  performed  by  collectives:  crews,  groups,  teams,  and  units.  In  military  and 
nonmilitary  activities,  the  skill  and  performance  of  collectives,  rather  than  the  individuals 
who  compose  them,  are  the  ultimate  objects  of  training.  In  this  perspective,  most,  if  not 
all,  training  of  individuals  is  intended  only  as  a  means  to  produce  successful— competent 
and  proficient— collectives. 

A  fair  question  for  research  is  whether  the  performance  quality  of  the  collective 
whole  (e.g.,  the  crew,  team,  or  unit)  differs  from  the  summed  capabilities  of  the  individu¬ 
als  who  comprise  it.  Collective  performance  ultimately  depends  on  the  performance  of 
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individuals.  We  should,  then,  expect  to  find  empirical  connections  in  the  research  litera¬ 
ture  between  the  performance  of  collectives  and  the  assessment  of  individuals  who  com¬ 
prise  them.  Several  studies  meet  these  expectations. 

Comrey  (1953)  found  that  individual  manual  dexterity  accounted  for  44  to 
5 1  percent  of  the  variance  in  the  performance  of  two-man  teams  performing  a  coopera¬ 
tive  manual  dexterity  task.  McGrath  and  Altman  (1966),  in  their  review  of  small-group 
research,  reported  that  individual  measures  of  job  experience  consistently  and  strongly 
accounted  for  collective  success.  O’Brien  and  Owens  (1969)  found  that  group  members’ 
abilities  accounted  for  about  34  percent  of  the  variance  in  their  productivity  on  highly 
coordinated  tasks.  Jones  (1974)  showed  that  individual  effectiveness  accounted  for  36  to 
81  percent  of  the  variance  in  the  success  of  tennis,  football,  baseball,  and  basketball 
teams.  Tziner,  Houser,  and  Hoffman  (1985)  determined  that  increases  in  the  ability  and 
the  motivation  of  the  individuals  comprising  three-member  tank  crews  significantly 
improved  their  performance  of  military  tasks.  These  and  other  studies  indicate  that  col¬ 
lective  performance  can,  to  some  degree,  be  predicted  from  individual  capabilities  when  a 
collective  task  engages  the  skills  and  knowledge  of  individual  members  and  these  skills 
and  knowledge  are  reliably  measured. 

On  the  other  hand,  the  amount  of  collective  performance  not  accounted  for  by 
assessments  of  individual  members  is  also  notable.  Reviews  by  O’Brien  and  Owens 
(1969),  Dyer  (1984),  and  Kahan  et  al.  (1985)  concluded  that  the  ability  to  predict  collec¬ 
tive  performance  from  individual  member  capabilities  is  substantially  reduced  when  sub¬ 
stantial  member  interaction  is  required.  Stout,  Salas,  and  Carson  (1994)  directly 
examined  the  impact  of  coordination  requirements  on  the  performance  of  two-person 
teams.  Interactive  processes  required  by  their  task  included  behaviors  such  as  providing 
information  in  advance,  making  plans,  asking  for  input,  assigning  tasks,  and  initiating 
assistance  to  others.  Stout  et  al.  found  that  coordination  ratings  for  the  teams  accounted 
for  26  percent  of  their  performance  variance  after  the  individual  abilities  of  the  two-team 
members  were  partialled  out. 

There  should  also  be  some  component  of  collective  performance  that  cannot  be 
accounted  for  by  individual-level  skills  but,  rather,  by  the  matches— the  complementa¬ 
tion— of  the  individual-level  skills  going  into  it.  This  component,  which  Roby  and  Lan- 
zetta  (1958)  called  the  “assembly  effect,’’  may  further  account  for  the  fact  that  the 
capabilities  of  individual  team  members  can  determine  only  a  part  of  team  performance. 
The  assembly  effect  is  understood  at  some  level  by  successful  coaches  and  military 
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commanders.  The  magnitude  of  the  assembly  effect  may  be  a  direct  measure  of  the 
amount  of  “teamness”  — the  amount  of  coordination  and  communication— required  to 
meet  the  team  objectives. 

The  training  issue  remains  unanswered.  Should  we  train  the  individual  or  the 
team?  Military  trainers  try  to  do  both,  but  the  point  at  which  training  for  collective  per¬ 
formance  must  pass  from  individual  training  to  training  within  the  collective  remains 
poorly  understood.  Perhaps  all  teaming  skills  can  be  trained  individually,  or  perhaps  indi¬ 
viduals  can  only  master  the  collective  skills  of  communication,  coordination,  and  timing 
by  training  within  teams.  The  answer  most  probably  depends  on  the  type  of  team  to  be 
trained. 

B.  RESIDENTIAL  AND  UNIT  TRAINING 

The  unique  characteristics  of  military  life  and  culture  require  newly  recruited 
military  personnel  to  undergo  some  early  training  to  prepare  them  for  positions  in  opera¬ 
tional  military  units.  Much  of  the  initial  transition  from  civilian  to  military  life  involves 
an  enculturation.  Despite  the  emphasis  on  military  courtesy  and  discipline  in  this  early 
training,  a  substantial  body  of  hard-core  skills  and  knowledge  has  to  be  mastered.  As  the 
training  proceeds,  it  increasingly  emphasizes  the  certification  of  individuals  for  specific 
military  occupational  specialties. 

Training  is  never  complete.  Continued  growth  in  skill  and  knowledge  is  required 
as  careers  advance.  Individuals  may  transfer  to  different  military  occupational  specialties. 
They  may  require  training  to  operate,  maintain,  or  deploy  specific  and  unique  systems 
found  in  their  operational  units.  They  may  train  to  prepare  for  unit- specific  missions. 
They  will  certainly  advance  in  rank  and  require  training  to  prepare  for  new  responsibili¬ 
ties  occasioned  by  their  growing  maturity,  competencies,  and  military  stature.  They  may 
receive  follow-on  training  from  their  units,  from  specialized  training  organizations  to 
which  they  return,  from  manufacturing  representatives,  or  from  mentors  in  on-the-job 
training.  The  distinction  of  residential  vs.  unit  training  concerns  whether  individuals  are 
trained  by  specialized  training  organizations  or  by  the  operational  units  to  which  they 
have  been  assigned. 

Either  form  of  training  (residential  or  unit)  can  be  accomplished  to  some  extent  by 
distance  or  distributed  learning  techniques.  That  is,  either  can  be  performed  without  the 
physical  presence  of  a  human  instructor.  This  training  can  be  accomplished  synchro¬ 
nously,  usually  through  either  one-  or  two-way  video  teletraining  (VTT)  supplemented  by 
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two-way  audio.  This  approach  transcends  locality  but  still  requires  students  to  assemble 
at  particular  times  and  places  in  classrooms. 

Distributed  training  can  also  be  performed  asynchronously,  often  using  one  form 
or  another  of  CBI  perhaps  supplemented  by  multimedia  or  “intelligent”  tutoring  capabili¬ 
ties.  It  can  be  delivered  by  mailed  media  such  as  videotape,  CD-ROM,  or  even  paper- 
based  correspondence  materials.  Notably,  it  can  be  delivered  through  the  Internet  over  the 
World  Wide  Web.  All  these  possibilities  for  asynchronous  training  transcend  both  local¬ 
ity  and  time.  Much  U.S.  DoD  training  is  now  being  converted  from  classroom-based, 
synchronous  training  to  asynchronous  training  through  the  use  of  instructional  technol¬ 
ogy- 

1.  Residential  Training  of  Individuals 

Residential  training  of  individuals  is  often  called  “schoolhouse  training.”  This 
type  of  training  uses  procedures  and  techniques  that  are  most  like  those  found  in  civilian 
education  and  training.  It  is  provided  by  military  schools  in  military  units  whose  primary 
mission  is  to  conduct  training.  It  is  intended  to  develop  the  skills,  knowledge,  and  atti¬ 
tudes  needed  to  certify  individuals  for  the  performance  of  specific,  identifiable  jobs.  It 
often  results  in  certification,  such  as  the  awarding  of  an  occupational  rating  (air  traffic 
controller,  military  policeman,  radar  repairman)  or  subspecialty  within  an  occupational 
rating  (parachute/airborne  qualification,  specific  language  ability,  specific  equipment 
qualification,  higher-rated  skill  level). 

2.  Unit  Training  of  Individuals 

Unit  training  of  individuals  typically  occurs  after  individuals  are  assigned  to 
operational  units.  This  training  is  typically  conducted  by  those  units— not  by  military 
training  commands.  Its  instructors  are  usually  individuals  who  are  also  assigned  to  the 
operational  unit.  It  usually  assumes  that  students  have  received  initial  skill  training,  that 
they  have  completed  much  of  the  transition  from  civilian  to  military  life,  and  that  unit 
training  should  build  on  their  initial  training  by  increasing  the  breadth  and  depth  of  their 
skills  and  knowledge.  Like  residential  training  for  individuals,  unit  training  of  individuals 
is  intended  to  prepare  individuals  to  perform  identifiable  jobs,  but  these  jobs  are  deter¬ 
mined  by  the  specific  billets  and  equipment  found  in  the  unit.  Unit  training  is  specifically 
designed  to  meet  unit  needs  and  probable  mission  requirements. 
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3.  Residential  Training  of  Collectives 

As  noted  previously,  nearly  all  military  operations  are  accomplished  by  collec¬ 
tives— by  individuals  acting  as  members  of  crews,  teams,  and  units.  Much  of  this  training 
is  viewed  as  the  responsibility  of  local  commanders  and  is  conducted  within  units.  Some 
of  it,  however,  is  conducted  outside  local  units  by  organizations  whose  primary  mission 
is  the  training  of  collectives.  For  example,  antisubmarine  warfare  (ASW)  teams  may  be 
taken  off  ships  and  trained  as  teams  in  land-based  simulators  intended  to  provide  inten¬ 
sive  practice  exercise  with  feedback  in  ASW  operations.  Another  prominent  example 
comes  from  the  U.S.  Army’s  National  Training  Center  (NTC),  which  brings  battalions 
and  brigades  to  Fort  Erwin,  an  instrumented  range  just  south  of  Death  Valley  in  Califor¬ 
nia,  to  engage  in  what  is  typically  two  weeks  of  exercises  against  a  permanent,  on-site 
opposition  force  that  functions  under  the  tactical  doctrine  of  probable  U.S.  adversaries. 
Skilled  operator-controllers  are  scattered  throughout  the  unit  being  trained  and  provide 
detailed  evaluative  feedback  to  all  members  of  the  unit  in  after-action  reviews  (AARs). 
Bolger  (1986)  provides  an  engrossing  description  of  NTC  operations. 

4.  Unit  Training  of  Collectives 

Historically,  the  unit  training  of  collectives  is  the  most  typical  form  of  military 
training.  Residential  schools  prepare  pilots,  cooks,  and  infantry  to  some  level  of  certified 
individual  proficiency;  however,  the  training  they  receive  in  units  (to  function  as  mem¬ 
bers  of  teams  and  crews,  with  the  specific  equipment,  jobs,  and  individuals  with  whom 
they  may  perform  military  operations)  is  a  prime  responsibility  of  every  unit  commander. 
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V.  SIMULATION 


Military  operations  put  at  risk  people  (military  and  civilian),  highly  sophisticated, 
expensive  equipment,  national  security,  and  international  relations.  We  need  to  prepare 
individuals  to  perform  these  operations  successfully  while  minimizing  the  dangers.  What 
they  must  learn  requires  the  application  of  knowledge  and  exercise  of  skills  that  cannot 
be  sufficiently  learned  in  lecture  halls. 

How  then  are  they  to  learn  and  practice,  with  adequate  feedback,  the  skills  and 
knowledge  they  need  to  perform  military  missions?  The  answer  must  be  found  in  simula¬ 
tion.  Heavy  reliance  on  instructional  simulation  is  characteristic  of  military  training. 

Simulated  environments  permit  training  of  a  nature  that  cannot  or  should  not  be 
attempted  without  simulation.  In  simulated  environments,  aircraft  can  be  crashed,  expen¬ 
sive  equipment  can  be  ruined,  and  lives  can  be  jeopardized  in  ways  that  range  from 
impractical  to  unthinkable.  Simulated  environments  also  provide  other  benefits  for 
training.  They  can  make  the  invisible  visible,  compress  or  expand  time,  and  reproduce 
events,  situations,  and  decision  points  over  and  over.  Simulation-based  training  is  not  a 
degraded  reflection  of  the  more  realistic  environments  we  would  prefer  to  use.  It  allows 
us  to  train  aspects  of  performance  that  would  otherwise  be  inaccessible.  It  offers  at  least  a 
partial  solution  to  Hiller’s  criterion  problem. 

A.  SIMULATION  TRAINING  FOR  INDIVIDUALS 

From  the  days  of  the  medieval  quintain  and  doubtless  before,  simulation  has  been 
prominent  in  conducting  military  training  and  assessing  the  readiness  of  individuals, 
crews,  teams,  and  units  to  perform  military  operations.  Today,  simulation  is  as  familiar  to 
mud-weary  soldiers  participating  in  field  exercises  as  it  is  to  commanders  maneuvering 
corps  of  computer-generated  forces  (CGF)  sweeping  across  vast,  electronic  plains  in  our 
war  colleges.  It  is  supported  by  devices  ranging  from  plastic  mock-ups  to  laptop  comput¬ 
ers  to  full-motion  aircraft  simulators  that  cost  more  than  the  aircraft  they  simulate. 

Some  rough  distinctions  among  models,  simulators,  and  simulation  is  useful. 
Models  provide  the  underlying  representation  of  inputs  and  outputs.  They  are  the  engines 
that  control  and  determine  the  responses  of  simulators  and  simulations  to  users. 


V-l 


Simulators  are  devices.  They  are  intended  to  represent  to  the  student  other  devices  or 
phenomena  likely  to  be  encountered  in  the  natural  world  and  are  used  to  produce  simula¬ 
tions  of  the  natural  world.  Simulations  are  the  products  of  simulators  and  their  underlying 
models.  A  simulation  is  a  system  of  models  and/or  simulators  that  represents  another 
system,  environment,  or  situation. 

B.  SIMULATION  AND  TRANSFER 

The  realism,  or  “fidelity,”  needed  for  simulation  to  perform  successful  assessment 
and  training  is  a  perennial  topic  of  discussion  (e.g.,  Hays  and  Singer,  1989).  Much  of  this 
discussion  responds  to  the  intuitive  appeal  of  Thorndike  and  Woodworth’s  (1901)  argu¬ 
ment  for  the  presence  and  necessity  of  identical  elements  to  ensure  the  successful  transfer 
of  that  which  is  learned  in  training  to  that  which  is  needed  on  the  job. 

Thorndike  and  Woodworth  suggested  that  such  transfer  is  always  specific  (never 
general)  and  is  keyed  to  either  substance  or  procedure.  More  recent  studies  of  transfer, 
such  as  the  widely  noted  paper  by  Gray  and  Orasanu  (1987)  who  remark  on  the  “surpris¬ 
ing  specificity  of  transfer,”  echo  this  point  of  view.  As  Holding  (1991)  points  out,  the 
identical  elements  theory  is  one  with  which  it  is  hard  to  argue.  It  seems  reasonable  to 
expect  task  elements  mastered  in  simulation  to  be  performed  with  some  appreciable 
degree  of  success  on  the  job. 

For  dynamic  pursuits  such  as  combat,  where  unique  situations  are  frequent  and 
expected,  the  focus  on  identical  elements  often  leads  to  an  insistence  on  maximum 
fidelity  in  simulations  used  for  assessment  and  training.  Because  we  do  not  know  pre¬ 
cisely  what  will  actually  happen,  we  assume  that  we  must  provide  as  many  identical  ele¬ 
ments  as  we  can.  This  prescription  would  suggest  a  viable  approach  if  fidelity  came  free, 
but  it  does  not. 

As  fidelity  rises,  so  do  costs  (e.g.,  Miller,  1974;  Holding,  1991).  High  costs  can  be 
borne  but  will  also  reduce  the  number,  availability,  and  accessibility  of  valuable 
resources  that  can  be  routinely  provided  for  assessment  or  training.  Therefore,  we  must 
reduce  costs  by  selecting  the  correct  degree  of  fidelity  needed  to  achieve  our  objectives. 
Notably,  the  choice  is  not  simply  high  fidelity  and  high  costs  vs.  low  fidelity  and  low 
costs.  The  optimal  choice  is  somewhere  in  between,  and  it  keys  on  careful  explication  of 
training  objectives.  The  processes  of  finding  this  optimal  choice  of  fidelity  based  on 
training  objectives  usually  cluster  under  the  issue  of  “selective  fidelity.” 
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Simulation  does  appear  to  work.  Improvements  in  the  quality  with  which  criterion 
tasks  are  performed  and  the  probability  that  they  will  be  performed  at  all  can  be  partially 
accounted  for  by  time  spent  in  simulation.  Evidence  of  this  utility  comes  from  many 
sources.  In  aircrew  training,  the  issue  keys  on  transfer:  Are  the  skills  and  knowledge  ac¬ 
quired  in  simulation  of  value  in  flying  actual  aircraft?  That  is,  do  they  transfer  from  one 
situation  to  the  other?  Many  attempts  to  answer  this  question  rely  on  transfer  effective¬ 
ness  ratios  (TERs)  (as  in  Roscoe  and  Williges,  1980).  These  ratios  can  be  defined  for 
pilot  training  as  follows: 


where  TER  =  transfer  effectiveness  ratio,  Ac  =  time  required  to  reach  criterion  perform¬ 
ance,  without  access  to  simulation,  As  =  time  required  to  reach  criterion  performance, 
with  access  to  simulation,  and  S  =  simulator  time. 

Roughly,  this  TER  is  the  ratio  of  aircraft  time  savings  to  the  expenditure  of  simu¬ 
lator  time.  It  tells  us  how  much  aircraft  time  is  saved  for  every  unit  of  simulator  time 
invested.  If  the  TER  is  small,  a  cost-effectiveness  argument  can  still  be  made  for  simula¬ 
tion  because  simulator  time  is  likely  to  cost  much  less  than  aircraft  time. 

Orlansky  and  String  (1977)  investigated  this  issue  in  an  often  cited  study.  They 
calculated  34  TERs  from  assessments  of  transfer  performed  from  1967  to  1977  by  mili¬ 
tary,  commercial,  and  academic  organizations.  The  TERs  ranged  from  -0.4  to  1.9,  with  a 
median  value  of  0.45.  Orlansky,  Knapp,  and  String  (1984)  also  compared  the  cost  of 
flying  actual  aircraft  with  the  cost  of  “flying”  simulators.  Very  generally,  they  found  that 
(1)  the  cost  of  operating  a  flight  simulator  is  about  one-tenth  the  cost  of  operating  a  mili¬ 
tary  aircraft;  (2)  an  hour  in  a  simulator  saves  about  half  an  hour  in  an  aircraft;  so  that  (3) 
use  of  flight  simulators  is  cost-effective  if  the  TER  is  0.2  or  greater. 

At  one  level,  this  finding  is  extremely  useful  and  significant.  However,  nothing  is 
simple,  and  a  few  caveats  may  be  in  order: 

•  As  Provenmire  and  Roscoe  (1973)  pointed  out,  not  all  simulator  hours  are 
equal.  Early  hours  in  the  simulator  appear  to  save  more  aircraft  time  than 
later  ones.  This  consideration  leads  to  learning  curve  differences  between 
cumulative  TERs  and  incremental  TERs,  with  diminishing  returns  best 
captured  by  the  latter. 

•  Transfer  is  not  a  characteristic  of  the  simulator  alone.  Estimates  of  transfer 
from  a  simulator  or  simulated  environment  must  also  consider  what  the 
training  is  trying  to  accomplish  (the  training  objectives).  This  issue  is  well 
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illustrated  in  a  study  by  Holman  (1979),  who  found  24  TERs  for  a  CH-47 
helicopter  simulator  ranging  from  2.8  to  0.0,  depending  on  which  training 
objective  was  under  consideration. 

•  There  is  an  interaction  between  knowledge  of  the  subject  matter  and  the 
value  of  simulation  alone.  Gay  (1986)  and  Fletcher  (1990)  both  found  that 
the  less  the  student  knows  about  the  subject  matter,  the  greater  the  need  for 
tutorial  guidance  in  simulation.  The  strategy  of  throwing  a  naive  student  into 
a  simulator  with  the  expectation  that  learning  will  occur  may  not  be  viable. 

•  The  operating  costs  of  aircraft  differ  markedly  and  will  create  quite  different 
tradeoffs  between  the  cost  effectiveness  of  training  with  simulators  and  with¬ 
out  them.  In  contrast  to  the  military  aircraft  considered  by  Orlansky,  Knapp, 
and  String,  where  the  cost  ratio  was  about  0.1,  Provenmire  and  Roscoe  were 
concerned  with  flight  simulation  for  the  Piper  Cherokee,  where  the  cost  ratio 
was  0.73. 

Other  caveats  may  occur  to  the  reader.  Specific  applications  deserve  specific  attention. 

The  value  of  simulation  is  not  limited  to  flight.  We  could  also  use  TERs  to  assess 
the  use  of  simulation  in  maintenance  training.  Time  to  effect  repairs  on  actual  equipment 
after  using  actual  equipment  in  training  could  be  compared  with  time  to  effect  the  same 
repairs  after  using  simulated  equipment  in  training.  However,  results  reported  from 
maintenance  training  assessments  rarely  report  time  to  repair.  Instead,  results  from  “con¬ 
ventional”  training  involving  lecture,  textbooks,  and  hands-on  experience  with  actual 
equipment  are  usually  contrasted  with  training  that  replaces  the  actual  equipment  with 
simulated  equipment.  In  these  cases,  effect  size,  which  is  the  standard  measure  from 
meta-analysis  used  to  quantify  and  combine  results  from  independent  and  separate 
assessment  studies,  can  generally  be  calculated  and  used  (Glass,  McGaw,  and  Smith, 
1981;  Hedges  and  Olkin,  1985). 

From  a  broad  review  of  interactive  multimedia  capabilities  used  for  simulation, 
Fletcher  (1997)  extracted  11  studies  in  which  simulated  equipment  was  used  to  train 
maintenance  technicians.  These  studies  compared  instruction  with  the  simulators  to 
instruction  with  actual  equipment,  held  overall  training  time  roughly  equal,  and  assessed 
final  performance  using  actual  (not  simulated)  equipment.  Over  the  1 1  studies,  the  use  of 
simulation  yielded  an  effect  size  (which  is  the  measure  of  merit  in  such  meta-analyses)  of 
0.4  standard  deviations,  suggesting  an  improvement  from  50th  percentile  to  about  66th 
percentile  achievement  among  students  using  simulation.  Operating  costs  using 
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simulation  were  about  40  percent  of  those  without  it  because  the  equipment  being  simu¬ 
lated  did  not  break  and  could  be  presented  and  manipulated  on  devices  costing  one  to  two 
orders  of  magnitude  less  than  the  actual  equipment  that  was  the  target  of  the  training. 

C.  SIMULATION,  COSTS,  AND  COST  EFFECTIVENESS 

Clearly,  we  would  like  to  take  account  of  costs  and  transfer  effectiveness.  We 
would  like  to  allocate  training  time  or  trials  between  simulation  and  “hands-on”  exercise 
with  the  intended  equipment  or  situation,  so  that  total  costs  to  train  are  minimized. 
Approaches  for  solving  this  problem  can  be  clustered  under  the  heading  of  “isoperfor¬ 
mance”  (Bickley,  1980;  Chronholm,  1985;  Morrision  and  Holding,  1990;  Jones  and 
Kennedy,  1996).  Basically,  the  idea  is  to  devise  a  function,  usually  depicted  as  an  isoper¬ 
formance  curve,  showing  every  point  where  different  combinations  of  training  inputs 
produce  equivalent  performance  outputs.  The  problem  is  then  simply  to  find  the  point  on 
the  curve  where  costs  are  minimized.  Of  course,  the  full  story  is  not  that  simple. 

As  suggested  by  discussions  of  the  engineering  of  instruction  (e.g.,  Woolf  and 
Regian,  2000),  the  outcome  of  a  training  system  is  typically  an  integrated  set  of  per¬ 
ceptual,  cognitive,  and  psychomotor  skills  with  supporting  knowledge  that  permits  an 
individual  to  perform  a  prescribed  set  of  tasks  or  a  job.  Some  of  these  skills  may  be  pre¬ 
sent  at  the  outset,  some  cannot  be  taught  until  the  individual  acquires  prerequisite  knowl¬ 
edge  and/or  skills,  and  some  performance  levels  may  be  beyond  the  practical  reach  of 
individual  learners.  Learning  is  further  complicated  by  the  proactive  and  retroactive 
effects  of  other  learned  skills,  learning  losses  that  may  differentially  affect  performance, 
individual  learning  styles,  and  individual  learning  rates.  Optimal  combinations  of  all 
these  factors  are  desirable  and  cost-effective  objectives  for  training  system  designers. 

These  combinations  might  be  sought  and  perhaps  found  through  an  extended  pro¬ 
gram  of  experiments  intended  to  identify  salient,  high-leverage  variables  and  assess  their 
effects  on  training  outcomes.  Such  an  approach  would  be  expensive,  time-consuming, 
and  disruptive.  Faced  with  these  real-world  constraints,  we  may  instead  develop  an 
empirically  based  model  of  the  training  system  that  incorporates  interrelationships  and 
analytical  tradeoffs  between  various  combinations  of  inputs,  the  costs  of  these  combina¬ 
tions,  and  the  training  outcomes  they  produce.  The  validity  of  such  a  model— and  its  out¬ 
puts— then  becomes  important  issue,  but  it  is  not  unreasonable  to  expect  solutions 
presented  by  the  model  to  be  worth  the  effort  to  develop  it. 
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As  suggested  previously,  isoperformance  relates  two  or  more  training  inputs  to  a 
training  outcome  held  at  some  prescribed  value  or  level.  It  is  generally  assumed  that  each 
input  by  itself  could  produce  the  desired-level  outcome,  although  some  inputs  could  con¬ 
tribute  uniquely  to  the  outcome,  necessitating  at  least  some  application  of  these  inputs. 
Different  amounts  of  the  separate  inputs  can  then  be  combined  to  produce  the  desired 
level  of  performance.  Isoperformance  identifies  all  combinations  of  the  given  inputs  that 
produce  a  given  output  level  of  performance.  This  identification  is  usually  done  on  a 
continuum,  which  produces  an  isoperformance  curve  in  the  case  of  two  input  variables 
and  an  isoperformace  surface  or  “hull”  in  the  multidimensional  case  of  three  or  more 
inputs. 

A  common  use  of  isoperformance  is  to  assess  different  combinations  of  simula¬ 
tion  and  actual  equipment  training,  as  shown  in  Figure  V-l.  Total  costs  are  shown  in  the 
upper  curve.  Performance  that  is  the  output  of  the  training  is  expected  to  be  the  same 
everywhere  on  the  total  cost  curve.  Total  costs  initially  decrease  as  simulator  time  is  sub¬ 
stituted  for  time  on  the  (presumably  more  expensive)  equipment  and  then  increase  as 
more  and  more  simulator  time  is  required  to  compensate  for  reductions  in  actual  equip¬ 
ment  time.  Costs  for  actual  equipment  time  start  in  the  same  place  as  total  costs  but  then 
decrease  monotonically  as  more  and  more  simulator  time  is  substituted  in.  Notably,  they 
never  reach  zero  because  we  assume  that  sooner  or  later  the  training  program  will  have  to 
include  some  time  or  trials  with  the  actual  equipment.  Costs  for  simulator  time  start  at 
zero  and  rise  monotonically  with  the  increasing  use  of  simulator  time  or  trials. 

The  cost-effective  solution  under  this  formulation  is  to  find  the  minimum  on  the 
upper,  total  costs  curve  and  use  it  to  allocate  training  time  or  trials  between  simulation 
and  the  real  equipment.  In  effect,  it  holds  performance  (or  effectiveness)  constant  and 
tries  to  minimize  costs.  Carter  and  Trollip  (1980)  illustrate  the  other  side  of  the  coin. 
They  used  what  Cronholm  (1985)  showed  to  be  a  mathematically  equivalent  approach  to 
devise  an  optimal  strategy  for  maximizing  performance  (or  effectiveness  again)  given 
fixed  costs. 

Morrison  and  Holding  concentrated  on  gunnery  training.  The  main  idea  was  to 
use  simulation  to  save  training  ammunition.  Other  examples  are  possible.  Bickley  (1980) 
focused  on  simulator  vs.  flight  time  in  the  Army’s  AH-1  helicopter.  Jones  and  Kennedy 
(1996)  discussed  the  application  of  isoperformance  functions  for  trading  off  personnel 
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Figure  V-1.  Components  of  the  Total  Cost  Function 
(Adapted  from  Morrison  and  Holding,  1990) 


aptitude  against  training  time.  Other  applications  can  be  expected  as  the  use  of  these 
functions  increases;  however,  the  serious  problem  of  collecting  appropriate  transfer  data 
remains  to  be  solved. 

Adequate  data  collection  to  perform  valid  isoperformance  analyses  — data  that 
show  all  combinations  of  training  inputs  that  produce  equivalent  performance  out¬ 
puts— can  easily  swamp  a  training  developer’s  budget.  One  well-precedented  solution  for 
this  sort  of  problem  is  to  substitute  the  judgment  of  experts  for  the  empirical  data  that  are 
needed.  Morrison  and  Holding  reviewed  these  possibilities  and  concluded  that  neither 
approach,  by  itself,  is  satisfactory.  Findings  from  empirical  study  are  sufficiently  valid 
but  prohibitively  expensive.  Expert  judgments  are  relatively  inexpensive  but  of  dubious 
validity.  The  answer  may  lie  somewhere  in  between.  We  may  be  able  to  increase  the 
validity  of  experts’  judgments  by  informing  and  shaping  them  with  limited  but  valid 
empirical  data.  Guidelines  for  implementing  such  an  approach  remain  to  be  developed. 
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VI.  SIMULATION  TRAINING  FOR  COLLECTIVES 


A.  SHARED  MENTAL  MODELS  -  AND  SIMULATIONS 

Training  for  collectives  has  recently  centered  on  the  issue  of  shared  mental  mod¬ 
els.  By  mental  models,  researchers  have  in  mind  “mechanisms  whereby  humans  are  able 
to  generate  descriptions  of  system  purpose  and  form,  explanations  of  system  functioning 
and  observed  system  states,  and  predictions  (or  expectations)  of  future  system  states” 
(Rouse  and  Morris,  1986,  p.  349).  Shared  mental  models  are  then  descriptions,  explana¬ 
tions,  and  predictions  that  the  members  of  a  group,  such  as  a  military  crew,  team,  or  unit, 
hold  in  common.  Cannon-Bowers,  Salas,  and  Converse  (1993)  suggest  that  the  presence 
and  validity  of  these  shared  models  determine,  to  an  appreciable  extent,  the  success  of 
group  operations. 

The  issue  may  extend  beyond  shared  mental  models  to  something  that  might  be 
called  shared  mental  simulations.  Over  the  past  30  years,  general  theories  of  perception 
and  learning  have  changed.  They  have  evolved  from  the  fairly  strict  logical  positivism  of 
behavioral  psychology,  which  emphasizes  the  study  of  directly  observable  and  directly 
measurable  actions,  to  what  may  be  called  cognitive  psychology.  Cognitive  psychology 
gives  more  consideration  to  internal,  less  observable  processes  that  are  assumed  to  medi¬ 
ate  and  enable  human  learning  and  to  produce  the  directly  observable  behavior  that  is  the 
subject  of  behaviorist  approaches. 

The  keynote  of  these  notions,  which  currently  underlies  our  understanding  of 
human  perception,  memory,  and  learning,  may  have  been  struck  by  Ulric  Neisser  (1967), 
who  stated,  “The  central  assertion  is  that  seeing,  hearing,  and  remembering  are  all  acts  of 
construction,  which  may  make  more  or  less  use  of  stimulus  information  depending  on 
circumstances”  (p.  10). 

Neisser  came  to  this  point  of  view  because  a  large  body  of  empirical  evidence 
showed  that  many  aspects  of  human  behavior,  such  as  seeing  and  hearing,  simply  could 
not  be  accounted  for  by  external  physical  cues  reaching  human  perceptors,  such  as  eyes 
and  ears.  Additional  processes,  including  an  internally  (one  might  say  cognitively)  gener¬ 
ated  analysis  by  synthesis  process,  had  to  be  posited  to  account  for  well-established  and 
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observable  human  abilities  to  detect,  identify,  and  process  physical  stimuli.  Such  a  proc¬ 
ess  requires  an  active  synthesis  of  the  environment  based  on  a  runnable  cognitive  model, 
or  simulation,  that  is  validated  or  modified  as  needed  by  cues  impinging  on  sensory  per- 
ceptors.  It  is  the  actively  evolving  cognitive  simulation,  not  the  stimuli  alone,  that  is  said 
to  account  for  what  the  individual  understands  about  the  environment. 

As  information  and  data  become  available  to  collectives,  they  appear  to  be 
absorbed  and  integrated  into  a  rapidly  evolving  collective  simulation  of  the  rapidly 
evolving  external  environment.  The  decisions  that  emerge  then  result  from  running  the 
shared  simulation  forward  under  various  scenarios  and  parameters  set  collectively  by  the 
collective  to  identify  optimized  courses  of  action.  Members  of  the  collective  must 
therefore  take  responsibility  for  the  correctness  of  their  own  models  and  for  the  ability  of 
others  to  share  them. 

How,  then,  might  we  develop  a  common  runnable  model  or  simulation  that  will 
be  shared  by  all  members  of  a  collective?  One  way  may  be  to  expose  them  to  a  full  a  set 
of  common  cues  that  are  as  characteristic  as  possible  of  the  environment  in  which  they 
must  operate  and  allow  them  freedom  to  explore  and  develop  their  own  accurate— and 
sharable— representations  of  that  environment.  Once  again,  simulation  appears  to  be 
called  for. 

B.  SIMULATION  OF  MILITARY  OPERATIONS 

Despite  the  paucity  of  Kirkpatrick’s  Level  4  assessments  and  the  presence  of 
Hiller’s  criterion  problem,  the  heavy  weight  of  historical,  anecdotal,  and  partial  databased 
evidence  suggests  that  military  training  contributes  appreciably  to  success  in  combat. 
Still,  our  evidence  remains  incomplete  and  indirect.  It  concerns  prepared¬ 
ness— “readiness”— rather  than  effectiveness.  It  accounts  poorly  for  morale  and  leader¬ 
ship.  It  tends  to  emphasize  process  (number  of  exercises  completed,  hours  flown,  miles 
driven)  rather  than  performance  (knowledge  and  skills  attained).  Commanders  and 
training  decision-makers  are  aware  of  these  issues.  They  have  been  vigorous  and  inven¬ 
tive  in  seeking  ways  to  assess  readiness,  training,  and  probable  effectiveness  in  per¬ 
forming  military  operations,  but  evidence  remains  partial,  indirect,  and  imprecise. 

One  area  on  which  many  agree  is  the  simulation  of  combat  engagements.  A  fre¬ 
quently  cited  finding  on  the  military  value  of  assessing  and  training  human  competence 
for  military  engagements  involves  the  impact  of  the  Navy’s  TOP  GUN  exercises 
(Gorman,  1990).  During  the  air  war  over  North  Vietnam,  roughly  1965-1973,  the  U.S. 


VI-2 


Navy  and  U.S.  Air  Force  flew  aircraft  of  comparable  capabilities.  In  fact,  many  of  the 
aircraft  used  were  exactly  the  same,  armed  with  the  same  weapons.  During  the  first 
4  years  of  air-to-air  combat,  the  Navy  and  the  Air  Force  yielded  an  identical,  and  disap¬ 
pointingly  low,  ratio  of  North  Vietnamese  to  U.S.  aircraft  lost:  2.2  to  2.4  North  Vietnam¬ 
ese  aircraft  for  every  U.S.  aircraft  downed  (as  shown  in  Figure  VI- 1  for  the  period 
1965-1968). 


Figure  VI-1.  Loss  and  Exchange  Ratios  in  U.S.  Air-to-Air  Engagements  1965-1973 

From  1968  to  1970,  there  was  a  halt  in  combat  operations  over  North  Vietnam. 
During  this  period,  the  Navy  initiated  a  training  program  using  simulated  engagements  to 
reduce  vulnerability  in  air-to-air  combat.  Navy  student  pilots  were  pitted  against  “enemy” 
pilots.  Other  Navy  pilots  trained  in  enemy  tactics  and  flying  MIG-type  aircraft.  Engage¬ 
ments  were  played  and  replayed  until  the  Navy  student  flyers  got  them  right. 

This  activity  was  perceived  to  have  significant  military  value,  but  the  question  of 
validity  remained.  Did  success  in  engagement  simulation  predict  success  in  combat 
operations?  The  question  was  answered  in  1970  when  the  air  war  resumed.  As 
Figure  VI- 1  suggests,  Navy  pilots,  still  flying  the  same  aircraft  as  their  Air  Force  coun¬ 
terparts  but  trained  using  engagement  simulation,  performed  about  six  times  better  than 
Air  Force  pilots,  whose  training  had  remained  unchanged.  As  the  figure  shows,  the  new 
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loss-exchange  ratios  were  2.4  for  Air  Force  pilots  and  12.5  for  Navy  pilots.  Given  these 
results,  the  Air  Force  adopted  a  similar  form  of  engagement  simulation  in  1974  for 
training  its  fighter  pilots. 

These  results  are  promising  because  they  suggest  that  success  in  simulation  pre¬ 
pares  individuals  and  collectives  for  real-world  military  operations.  However,  the 
approach  is  costly.  It  requires  use  of  actual  equipment  (airplanes  in  the  TOP  GUN  exer¬ 
cises,  but  tanks  on  the  ground  and  ships  at  sea  in  other  engagement  simulations).  Large 
numbers  of  individuals  are  also  needed  (e.g.,  to  provide  umpires  and  logistics  support)  for 
the  training.  The  exercise  range  and  equipment  used  are  instrumented  with  position 
locators,  lasers,  and  laser  sensors  to  provide  participants  sufficiently  accurate  feedback  on 
their  performance  so  they  can  diagnose  it  and  improve  it. 

These  costs  reduce  physical  accessibility,  opportunities  to  train  and  assess  per¬ 
formance,  and  availability  of  feedback.  For  instance,  Army  tactical  teams  at  large  unit 
levels,  such  as  corps  and  division,  train  only  about  twice  a  year  because  of  high  costs  and 
time  demands  (Garlinger  and  Fallesen,  1988).  Less  expensive  means  are  needed  to  sup¬ 
port  readiness  assessment.  Networked  simulation  provides  one  such  possibility. 

C.  TACTICAL  ENGAGEMENT  SIMULATIONS  (TESs) 

Today,  everything  short  of  combat  engagements  in  which  determined  combatants 
oppose  each  other  is  viewed  as  simulation.  Different  forms  of  simulation  are  not  viewed 
as  inherently  inferior  or  superior  to  each  other.  Their  value  keys  on  the  objectives  moti¬ 
vating  their  use.  Defense  planners  distinguish  among  three  forms  of  simulation:  live,  con¬ 
structive,  and  virtual  (Gorman,  1991). 

Live  simulation  involves  experience  in  the  field  (or  in  the  air  or  at  sea).  Actual 
equipment  is  operated  on  ranges  that  are  often  instrumented  to  record  relevant  activities. 
Constructive  simulation  is  best  exemplified  by  computerized  war  games.  Scenarios  are 
established,  and  parameters  are  set  and  applied  by  participants  who  step  back,  let  the 
computers  play  out  the  engagement,  and  then  return  to  observe  the  results.  Much  there¬ 
fore  depends  on  the  validity  of  the  underlying  model  that  controls  the  simulation.  Virtual 
simulation  is  in  between.  It  involves  manned  simulators  that  are  linked  together  and  that 
engage  each  other  on  common,  electronically  generated  terrain. 

The  relationships  among  all  three  forms  of  simulation  are  worth  noting  (see 
Figure  VI-2).  This  figure  suggests  the  following: 
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Constructive 
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Figure  VI-2.  Relationship  of  Three  Forms  of  Simulation  to  Each  Other 
and  to  Operational  Effectiveness 

Each  form  of  simulation  can  train  some,  but  not  all,  aspects  of  military 
operations.  No  form  of  simulation  trains  everything. 

There  are  aspects  of  operational  effectiveness  that  none  of  these  forms  of 
simulation  can  train. 

Some  aspects  of  operational  effectiveness  are  trained  by  more  than  one  form 
of  simulation. 

Some  aspects  of  operational  effectiveness  are  uniquely  trained  by  each  of 
these  forms  of  simulation.  Put  in  very  direct  terms,  no  one  form  of  simulation 
suffices  as  a  criterion  for  the  training  provided  by  another.  For  instance, 
using  live  simulation  to  validate  training  performed  in  virtual  simulation 
without  reference  to  the  specific  aspects  of  effectiveness  being  trained  is 
misleading. 

A  related  point  is  that  some  aspects  of  operational  effectiveness  are  better 
trained  by  one  form  of  simulation  than  another.  For  instance,  maneuver  may 
be  better  trained  in  virtual  simulation  than  in  live  field  exercises,  where 
maneuvers  may  have  to  avoid  constraints  presented  by  highways,  buildings, 
or  shipping  lanes  that  would  be  ignored  during  combat.  Troop  leadership 
skills  may  be  better  trained  in  the  rain,  snow,  or  desert  heat  of  field  exercises 
than  in  the  air-conditioned  buildings  used  for  virtual  and  constructive 
simulation. 
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Trainers  may  actually  be  able  to  give  something  back  in  this  area.  Trainers  are 
beginning  to  embrace  emerging  technologies  that  may  be  of  interest  to  those  concerned 
with  personnel  assessment.  Among  these,  simulation  is  most  prominent.  It  may  allow  us 
to  collect  data  on  the  absolute  rather  than  relative  standing  of  individuals  performing  a 
sample  of  the  whole  job,  with  all  necessary  tasks  integrated  into  the  measurement. 
Among  simulation,  networked  simulation  seems  to  rise  to  the  fore. 

D.  NETWORKED  SIMULATION 

Virtual,  networked  simulation  was  originally  developed  for  training  applications 
and  was  intended  to  improve  the  fighting  performance  of  crews,  teams,  and  units  (Alluisi, 
1991;  Miller  and  Thorpe,  1995).  The  essential  ingredient  that  networked  simulation 
brings  to  the  preparation  of  collectives  is  not  so  much  initial  instruction  (although  some 
can  be  provided),  but  an  opportunity  for  accessible,  frequent,  and  realistic  practice  with 
substantive,  understandable,  and  relevant  feedback.  As  its  principal  architect,  Jack 
Thorpe  (1987)  stated,  “This  [concern  with  practice]  emphasizes  what  we  already  know 
about  how  a  team  achieves  mastery  of  its  art,  be  it  a  sports  team,  an  orchestra,  an  oper¬ 
ating  room  team,  or  a  combat  team:  Massive  amounts  of  practice  are  demanded.  There  is 
no  substitute”  (p.  493). 

Networked  simulation  consists  of  modular  objects  intended  to  simulate  engaged 
entities.  Typical  entities  are  combat  vehicles,  such  as  tanks,  helicopters,  and  aircraft,  but 
the  entities  may  be  anything  relevant,  including  bridges,  buildings,  dismounted  infantry, 
and  engineered  obstacles.  Simulators  displaying  these  entities  may  be  geographically 
located  anywhere  because  they  are  modular  and  all  share  a  common  model  of  the  situa¬ 
tion  and  its  terrain.  In  a  networked  simulation,  a  tank  crew  in  a  simulated  tank  in  Ger¬ 
many  can  call  for  support  from  simulated  aircraft  in  Nevada  because  they  are  being 
attacked  by  a  helicopter  simulator  located  in  Alabama. 

Each  entity,  along  with  hundreds  of  others,  is  connected  to  the  network.  If  the 
simulated  combat  vehicles  encounter  allied  vehicles  on  the  digital  terrain,  they  can  join 
together  to  form  a  larger  combat  team  and  undertake  a  mission  with  all  the  problems  of 
command,  control,  communications,  coordination,  timing,  and  so  forth  that  such  activity 
presents  to  tactical  teams.  If  they  encounter  enemy  vehicles,  they  can  fight— engage  in 
force-on-force  combat  engagements  in  which  the  outcome  is  determined  solely  by  the 
performance  of  the  individuals,  crews,  teams,  and  units  involved.  No  umpires,  battle 
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masters,  or  other  outside  influences  are  expected  or  permitted  to  affect  the  outcome  of  a 
networked  simulation  engagement  once  it  begins. 

In  terms  used  by  Wagner  et  al.  (1977),  networked  simulation  focuses  on  emergent 
rather  than  established  task  situations.  There  is  no  training  “intelligence”  in  the  core  of 
this  technology.  Its  use  depends  on  the  intentions  of  the  user. 

The  individual  members  of  crews,  teams,  and  units  who  use  networked  simulation 
are  assumed  to  be  proficient  in  their  individual  skill  specialties.  They  are  expected  to 
know  how  to  drive  tanks,  read  maps,  fly  airplanes,  fire  weapons,  and  so  on  at  some 
acceptable  threshold  of  proficiency  before  they  begin  networked  simulation  exercises. 
Moreover,  the  commanders  of  these  crews,  teams,  and  units  are  expected  to  possess  some 
basic  academic  knowledge  and  practical  skills  in  the  command  and  control  of  their  col¬ 
lectives.  They  are  expected  to  know  at  some  rudimentary  level  how  to  maneuver,  use  ter¬ 
rain  in  a  tactically  appropriate  manner,  fly  helicopters,  create  and  overcome  engineered 
obstacles,  and  so  forth.  As  Dyer’s  (1984)  extensive  review  suggests,  these  prerequisite 
capabilities  are  essential  for  successful  team  training.  Networked  simulation  focuses  on 
collective  rather  than  individual  training. 

All  the  digital  communication  packets  used  to  control  networked  simulation  can 
be  recorded.  With  some  exceptions  implemented  to  reduce  network  communications  traf¬ 
fic,  entities  currently  issue  three  to  five  packets  per  second.  Actions  undertaken  in  net¬ 
worked  simulation  can  then  be  recorded  in  extensive  detail  for  later  analyses  and  replay. 

Characteristics  of  networked  simulation  especially  useful  in  collective  training 
are: 

•  Focus  on  groups.  Networked  simulation  is  intended  to  improve  the 
performance  of  groups  (i.e.,  crews,  teams,  and  units). 

•  Physical  dispersion  of  participants.  Networked  simulation  allows 
participants  who  are  geographically  dispersed  to  engage  and  exercise  with 
one  another,  without  the  expense  and  administrative  complication  required  to 
bring  them  into  physical  proximity.  Their  dispersion  is  constrained  only  by 
the  physical  limits  of  the  communication  network  that  connects  them. 

•  Real-time  responses.  The  technology  that  underlies  networked  simulation  is 
optimized  for  real-time  responsiveness.  It  is  particularly  useful  in  simulating 
environments  that  demand  responses  to  time-sensitive  challenges  that  must 
be  prioritized  and  met  as  rapidly  as  possible. 
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•  Emergent  task  environments.  Networked  simulation  was  originally 
designed  for  tasks  and  activities  that  could  not  be  prespecified  in  any 
deterministic  fashion.  These  tasks  evolve  rapidly  over  time  and  in  response  to 
actions  taken  in  the  simulated  environment.  Communication  and  coordination 
within  and  between  individuals,  crews,  teams,  units,  and  commanders  are 
free  and  uncontrolled.  Outcomes  are  determined  only  by  the  participants’ 
decisions  and  actions. 

•  Accessibility  of  performance  data.  Networked  simulation  allows  massive 
amounts  of  performance  data  to  be  recorded  and  then  easily  retrieved, 
viewed,  and  assessed.  Because  all  network  packets  can  be  recorded, 
performance  archives  can  all  be  played  and  replayed  using  the  same  visual, 
simulated  environment  in  which  they  were  recorded  but  viewed  from  any 
vantage  point  desired.  This  capability  for  visual  and  highly  detailed  replay 
provides  AARs  in  which  actuality —rather  than  rhetoric  or  uncertain 
observation— drives  feedback  and  adjudicates  assessment. 

•  Affordability.  Networked  simulation  is  intended  to  be  accessible  and 
economically  affordable. 

•  Provisions  for  realism.  The  environments  supported  by  networked 
simulation  increase  some  dimensions  of  realism  by  allowing  human 
performance  to  determine  outcomes,  placing  fewer  constraints  on  force-on- 
force  operations,  reducing  the  quantity  and  impact  of  artificial  conditions 
imposed  by  field  activities,  and  increasing  the  freedom  to  undertake 
operations  that  would  otherwise  be  too  dangerous  or  costly. 

•  Interaction  with  many  entities.  Networked  simulation  allows  its 
participants  to  interact  with  a  large  number  of  entities  that  must  respond  in  a 
realistic  fashion  to  their  actions.  These  entities  can  be  manned,  computer 
generated  but  human  controlled,  or  completely  computer  generated  and 
controlled.  The  capacity  for  computer  generation  and  control  allows  the 
number  of  entities  involved  to  become  sufficiently  large  to  meet  objectives 
that  would  otherwise  be  unaffordable  and  unattainable. 

Networked  simulation  can  currently  represent  three  types  of  entities: 

1 .  Manned.  The  principal  entities  are  manned,  or  crewed,  simulated  combat 
vehicles  — tanks,  helicopters,  and  aircraft.  All  their  key  functions  are  directly 
controlled  by  humans. 

2.  Automatic.  Some  entities  are  strictly  automatic.  Once  established  or  set  into 
motion,  no  human  intervention  (by  individuals,  crews,  teams,  or  units)  can 
affect  their  operation.  Examples  of  automatic  entities  in  early  versions  of  this 
technology  were  supply  vehicles,  which  were  requested  and  then  after  an 
appropriate  (i.e.,  realistic)  period  of  time  would  appear  at  the  place  requested. 
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They  did  not  travel  the  terrain  and,  therefore,  could  not  be  attacked  during 
transit  in  early  versions  of  networked  simulation. 

3.  Semi-automatic.  The  semi-automatic  entities  (mainly  vehicles)  of  the  semi¬ 
automatic  forces  (SAFOR)  are  among  the  most  technically  interesting, 
innovative,  and  challenging  aspects  of  networked  simulation.  These  entities 
are  indirectly  but  not  directly  manned,  or  crewed.  Human  control  is  required 
for  their  operation  at  some  level  above  that  of  crews  — a  platoon  commander 
controls  a  platoon  of  unmanned  vehicles,  a  company  commander  controls  a 
company  of  unmanned  vehicles,  and  so  forth— and  the  entities  must  act  and 
react  in  a  realistic  fashion.  They  must  behave  as  they  would  if  they  were 
manned.  Humans  on  the  electronic  battlefield  should  not  be  able  to  tell  the 
difference  between  manned  and  SAFOR  entities. 

Although  networked  simulation  was  originally  developed  for  training,  it  is  finding 
applications  in  many  other  military  activities.  It  is  being  used  in  the  design,  development, 
and  acquisition  of  materiel  and  systems.  For  instance,  we  can  include  different  perform¬ 
ance  capabilities  proposed  for  military  systems  in  force-on-force  engagements  to  deter¬ 
mine  if  the  difference  they  make  in  combat  effectiveness  is  worth  their  cost.  Once  proto¬ 
types  are  built,  preparation  for  proposed  tests  can  be  rehearsed  and  fine-tuned  using 
networked  simulation  before  beginning  trials  in  the  field.  Quick-strike  missions  can  be 
repeatedly  rehearsed  in  networked  simulation  using  digitized  terrain  of  the  area  where  the 
military  operations  are  likely  to  take  place,  simulation  of  the  allied  forces  likely  to  be 
found  there,  and  simulators  locally  positioned  with  the  military  units  involved  before  they 
must  be  physically  marshaled  together. 

Networked  simulation  appears  to  hold  considerable  promise  for  civilian  applica¬ 
tions.  Among  the  possibilities  discussed  are  (Fitzsimmons  and  Fletcher,  1995): 

•  Training  for  crews,  teams,  and  units.  Cannon-Bowers,  Salas,  and  Converse 
(1992)  documented  the  requirement  to  train  crews,  teams,  and  units  in 
nonmilitary  settings  by  citing  examples  of  21  such  groups  commonly  found 
outside  the  military.  These  include  quality  circles,  management  teams, 
maintenance  crews,  product-development  teams,  cockpit  crews,  surgery 
teams,  negotiating  teams,  instructor  teams,  and  athletic  teams.  This  list  could 
easily  be  augmented  by  the  inclusion  of  firefighting  teams,  well-drilling 
crews,  police  Special  Weapons  and  Tactics  (SWAT)  teams,  ship  crews, 
disaster  emergency  teams,  and  ground-air  control  teams.  Nearly  all  of  these 
noncombat  crews,  teams,  and  units  must  exercise  group  coordination  and 
communication  skills  to  meet  emergent  situations  with  time-sensitive 
demands. 


VI-9 


Networked  simulation  may  also  be  key  to  make  workplace  training 
accessible  and  affordable.  This  need  is  especially  great  in  small-  to  medium¬ 
sized  firms,  which  have  few  resources  to  devote  to  producing  and 
implementing  the  training  and  lifelong  learning  their  workers  need  (U.S. 
Congress  OTA,  Worker  Training ,  1990).  This  need  is  also  strongly  felt  by 
workers  who,  on  their  own,  are  attempting  to  improve  their  skills  or  transfer 
their  skills  to  new  areas  of  work.  Teamwork,  collaboration  to  solve 
motivating  and  job-relevant  problems,  access  to  subject  matter  experts 
(SMEs)  and  monitors,  and,  perhaps  most  importantly,  development  of 
learning  communities  in  our  workplaces  can  evolve  from  applications  of 
networked  simulation  used  to  meet  workforce  training  requirements. 

•  School-to-work  transition.  Transitions  from  school  to  work  and 
occupational  choices  are  often  haphazard  and  uninformed.  Students  need  to 
understand  and  directly  experience  the  responsibilities  and  the  rewards  of  the 
workplace  and  the  “look  and  feel”  of  activity  in  specific  occupations. 
Through  networked  simulation,  students  can  enter  into  realistic  workplace 
environments  with  their  fellow  students  or  with  workers  who  have 
accumulated  lifelong  experience  in  these  environments.  They  can  experience 
in  the  virtual  world  the  demands  and  workday  challenges  of  entry-level 
positions  along  with  the  eventual  satisfactions  and  rewards  of  many  different 
occupations  before  committing  to  the  training  and  other  preparation  they 
require  for  real-world  participation.  The  school-to-work  transitions  that  have 
been  the  target  of  much  recent  study  and  legislation  will  be  eased 
considerably  if  students  can  readily  access  realistic  experience  with  the 
variety  of  workplaces  to  which  they  might  aspire. 

•  Education.  Students  in  formal  educational  settings  are  seldom  required  to 
engage  in  real-time,  collaborative  interaction  as  members  of  problem-solving 
teams  despite  the  instructional  value  of  these  experiences.  Teachers  depend 
on  “chalk  and  talk”  to  convey  information,  and  students  are  viewed  more  as 
passive  recipients  of  learning  than  as  active  participants  in  a  teaching¬ 
learning  process.  Classrooms  continue  to  operate  as  isolated  entities  that 
provide  few  opportunities  for  networking  or  communication  with  the  outside 
world.  Educational  applications  for  networked  simulation  are  hard  to  find  in 
practice,  although  they  are  not  difficult  to  imagine. 

Despite  this  state  of  affairs,  today’s  educators  increasingly  emphasize  the 
value  of  collaborative  problem  solving,  project-based  learning,  and  learning 
by  doing.  New  applications  of  technology  in  education,  especially  networked 
simulation,  can  support  the  inclusion  of  more  group-oriented,  concrete,  and 
motivating  instructional  approaches.  The  most  promising  of  these 
applications  requires  students  to  collaborate  in  solving  problems.  The 
students  do  not  learn  for  the  purpose  of  gaining  abstract  benefits  in  the 
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indefinite  future  but  for  more  practically  obvious  and  immediately 
motivating  rewards.  Students  can  collaborate  with  others  in  geographically 
dispersed  locations,  satisfy  real-time  constraints  for  coordination  and 
response,  access  specialized  information  in  remote  digital  libraries,  perform 
experiments  using  virtual  laboratories  and  laboratory  equipment  that  outstrip 
locally  available  resources,  and  interrogate  SMEs  whose  specialized 
knowledge  they  need  to  solve  the  problems. 

E.  AFTER- ACTION  REVIEW  (AAR) 

A  particularly  interesting  practice  that  has  grown  out  of  the  use  of  TESs  in  mili¬ 
tary  training  is  the  AAR  process.  AARs  are  discussions  held  after  a  simulation  exercise 
and  are  usually  led  by  a  controller  or  trainer  from  the  exercise.  They  are  intended  to 
identify  and  discuss  what  happened  in  the  exercise,  why  it  happened,  and  how  perform¬ 
ance  of  the  participants  might  be  improved. 

Notably,  these  AARs  do  not  consist  of  performance  critiques  by  the  discussion 
leader.  As  practiced  by  the  U.S.  military,  all  discussion  of  what  happened,  why  it  hap¬ 
pened,  and  how  well  it  was  done  is  expected  to  arise  from  those  who  were  participants  in 
the  exercise,  not  from  controllers  or  trainers.  Further,  the  discussion  leader  and  partici¬ 
pants  work  together  to  ensure  that  their  AAR  proceeds  in  an  objective  and  nonpunitive 
atmosphere  of  cooperation  intended  to  improve  future  performance. 

Many  cultures  outside  the  United  States  find  this  model  of  review  difficult  to 
implement.  Points  in  U.S.  AARs  concerning  strengths  and  weaknesses  are  raised  frankly 
and  may  come  as  freely  from  subordinates  as  from  superiors.  Other  cultures  with  more 
stringent  class  distinctions  [e.g.,  those  between  officers,  noncommissioned  officers 
(NCOs),  and  enlisted;  between  lead  pilots,  wingmen,  and  ground  controllers;  or  between 
different  branches  (e.g.,  infantry,  armor,  air  defense)  of  the  armed  Service(s)  involved] 
find  this  process  too  candid  and  freely  discursive  to  implement.  Still  other  cultures  (e.g., 
those  in  which  “saving  face”  is  a  sensitive  and  critical  matter,  those  in  which  collective 
training  performance  is  tightly  bound  to  the  career  advancement  of  commanders,  or  those 
in  which  the  free  exchange  of  information  is  so  rarely  practiced  that  participants  simply 
do  not  understand  how  to  do  it)  also  find  implementation  of  the  U.S.  AAR  process  to  be 
difficult,  if  not  impossible.  Cultures  that  can  overcome  these  difficulties  find  payoff  from 
this  AAR  process  to  be  substantial.  It  stands  as  a  powerful  tool  for  improving  military 
performance  and  likelihood  of  success  in  performing  military  operations. 
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Further,  and  as  Morrison  and  Meliza  (1999)  demonstrate,  the  U.S.  AAR  process  is 
well  rooted  in  both  basic  and  applied  research.  It  represents  a  culmination  of  findings 
drawn  from  areas  such  as  intrinsic  and  extrinsic  feedback;  performance  measurement  and 
self  assessment;  transfer  of  training,  memory  aiding,  problem  solving,  decision  making, 
and  mental  models;  studies  of  group  processes;  social  facilitation,  identity,  and  cohesion; 
research  on  communication  theory  and  techniques;  guided  discovery  learning;  and  expe¬ 
riential  learning,  cooperative  learning,  and  general  systems  concepts.  Some  of  this 
research  was  applied  directly  to  develop  the  AAR  process,  and  some  was  used  to  support, 
focus,  and  rationalize  techniques  that  were  already  being  adopted  on  the  basis  of  intui¬ 
tion,  common  sense,  and  25  years  of  practice. 

Education  and  training  outside  the  military  also  review  the  processes  and  out¬ 
comes  of  simulation,  instructional  games,  and  other  interventions  involving  groups, 
crews,  staffs,  and  teams.  As  well-practiced  and  thoroughly  rationalized  procedures,  the 
AARs  conducted  by  U.S.  military  organizations  may  serve  as  a  model  for  improving  the 
productivity,  competitiveness,  and  efficiency  of  many  organizations  beside  those  con¬ 
cerned  with  military  operations. 
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VII.  CONCLUSIONS 


Military  training,  despite  or  perhaps  because  of  its  unique  concerns,  has  its  own 
perspectives  and  emphases,  but  many  of  these  may  be  of  interest  outside  the  military 
training  community. 

A.  TRAINING  TECHNOLOGY 

In  one  frequently  cited  study,  Bloom  (1984)  reported  on  a  series  of  experiments 
that  suggested  individual,  one-on-one  tutoring  produces  achievement  that  is  about  two 
standard  deviations  greater  than  that  produced  by  classroom,  one-on-many  instruction. 
Like  nearly  all  other  forms  of  training,  military  training  can  rarely  afford  one  instructor 
for  every  student.  It  has  looked  to  technology  to  supply  the  individualization  of  pace, 
sequence,  content,  style,  and  difficulty  that  might  account  for  Bloom’s  two  sigma  gap  and 
help  solve  Bloom’s  two  sigma  problem. 

This  interest  predates  Bloom’s  investigations.  It  started  with  funding  for  CBI  in 
the  mid-1950s  (Fletcher  and  Rockway,  1986)  and  continues  through  the  present  with  the 
current  ADL  initiative  intended  to  create  web-available  instructional  objects  that  are 
reusable  in  courseware  development,  portable  across  technology  platforms,  durable 
across  system  software  versions,  and  accessible  to  all  stakeholders. 

B.  INSTRUCTION  AS  ENGINEERING 

Training  objectives  are  likely  to  be  more  identifiable,  better  defined,  and  less 
negotiable  than  those  in  education.  As  a  consequence,  training  design  and  development 
principles  are  more  concerned  with  producing  predictable  results  than  with  the  art  of 
classroom  presentation.  Training  elicits  much  creativity,  but  this  creativity  is  more  likely 
to  be  found  in  the  inventiveness  of  design  required  to  produce  specific  outcomes  reliably 
and  cost  effectively  than  in  the  art  of  instructional  delivery. 

Military  training  is  particularly  concerned  with  enhancing  its  capabilities  to  engi¬ 
neer  reliable  training  outcomes.  Such  an  approach  was  infeasible  before  the  availability 
and  application  of  controlled  presentations  provided  by  instructional  technology.  Means 
of  linking  specific  design  decisions  to  specific  training  outcomes  are  now  being 
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developed  for  military  training.  They  may  interest  anyone  concerned  with  reliable,  pre¬ 
dictable  production  of  human  capabilities  and  competencies  through  training. 

C.  TRAINING  PRODUCTIVITY 

Training  productivity  is  not  a  matter  of  indifference  outside  the  military.  How¬ 
ever,  the  military,  which  assumes  a  responsibility  to  train  its  people  for  their  duties  from 
enlistment  through  retirement,  has  concentrated  on  developing  techniques  and  principles 
to  increase  training  productivity.  The  military  places  few  restrictions  on  disseminating  its 
findings.  In  fact,  in  many  cases,  actively  seeks  to  advance  the  state  of  training  practices 
by  promulgating  them  throughout  the  military  training  community. 

Limitations,  as  noted,  reside  in  the  lack  of  incentives  for  spreading  these  tech¬ 
niques  and  principles  beyond  that  community.  Such  information  remains  available 
through  DTIC  and  NTIS.  The  emphasis  in  military  training  on  developing  techniques  and 
procedures  for  examining  the  cost  effectiveness  of  different  training  approaches  and 
devising  optimal  combinations  of  these  approaches  to  achieve  training  objectives  may  be 
of  interest  to  training  and  education  communities  elsewhere. 

D.  COLLECTIVE  PERFORMANCE 

Most  military  operations  and  most  human  activities  are  performed  in  the  context 
of  collectives  — crews,  groups,  teams,  and  units.  The  extent  of  emphasis  on  collective 
training  is  perhaps  unique  in  the  military.  Most  academic  research  on  human  learning, 
performance,  and  cognitive  processes  centers  on  individuals.  Much  research  that  is  con¬ 
cerned  with  the  learning,  performance,  and  cognitive  processes  of  collectives  is  accom¬ 
plished  to  support  military  training.  It  has  produced  results  that  may  be  of  general 
interest.  Techniques  for  developing  shared  mental  models,  use  of  simulators  and  simula¬ 
tion  to  train  collectives,  processes  for  conducting  AARs,  and  measures  of  the  compe¬ 
tence,  productivity,  and  “readiness”  of  collectives  should,  among  other  developments, 
interest  those  in  training  communities  well  beyond  the  military. 

E.  RESEARCH  AND  DEVELOPMENT  (R&D) 

A  point  was  made  of  the  extent  to  which  the  military  invests  in  R&D  for  training. 
It  could  be  and  has  been  argued  that  the  major  source  of  training  R&D  in  the  United 
States  is  military  R&D.  Still,  the  amount  invested  by  the  DoD  is  small  relative  to  the  size 
of  the  enterprise  it  supports.  The  proportion  of  R&D  funding  to  the  total  DoD  training 
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enterprise  is  about  0. 1-0.2  percent.  This  contrasts  with  R&D  in  support  of  other  areas  of 
military  activity,  where  similar  proportions  of  effort  range  from  17  to  30  percent.  The 
contribution  of  military  R&D,  proportionally  small  though  it  may  be,  is  nationally 
important  and  worthy  of  national  attention. 

F.  TRADEOFFS  AND  TRAINING  AS  A  SUBSYSTEM 

A  point  belabored  in  this  document  is  that  training  is  a  means  to  an  end.  Training 
is,  however,  a  subsystem  of  a  more  comprehensive  system  established  to  ensure  the  pro¬ 
duction  and  supply  of  human  competence. 

Training  procedures  and  policies  should  not  be  set  in  a  vacuum.  The  full  system 
includes  personnel  selection  for  military  service,  classification  of  individuals  for  specific 
jobs,  assignments  to  career-building  responsibilities,  training,  human  factoring  and  ergo¬ 
nomic  design,  job  design,  and  supply  and  logistics  procedures.  All  these  subsystems 
interact.  Higher  standards  for  selection  and  classification  will  reduce  training  require¬ 
ments.  Training  can  compensate  for  poor  ergonomic  design,  refined  classification  proce¬ 
dures  can  make  up  for  recruiting  difficulties,  and  so  forth. 

The  evolution  and  development  of  military  training  will  continue  because  it  is  a 
national  priority.  Its  influence  on  training  elsewhere  has  been  substantial  but  indirect.  It 
will  continue  to  offer  much  in  the  way  of  techniques  and  technology  to  all  training  and 
education  activities. 
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